acceptodds
Under review as a conference paper at ICLR 2027

Beyond Task Success: A Theory of Representation Reuse

Abstract

A learned representation is useful beyond its source task only if later tasks can make use of what the encoder has already learned. This matters most when the new task has limited supervision. Source-task performance, however, gives no guarantee that the representation itself will remain useful once the original head is removed. We study representation reuse with respect to a specified family of future tasks. For such a family, we define the task-relevant state as the coarsest deterministic state sufficient for all of its tasks. A larger task family can require a finer state, and losing a required distinction makes at least one future task unrecoverable on a target support containing that conflict. Exact reuse also depends on how the retained information is presented to the downstream rule. We separate this into structural accessibility, which concerns whether the required distinctions remain recoverable, and coordinate consistency, which concerns whether those distinctions can be interpreted by the same downstream rule across samples. Both failure modes appear separately in the experiments. Reuse improves as task-relevant distinctions become more accessible. Reversible sample-dependent coordinate changes can sharply reduce reuse even when the representation remains recoverable, and inverting the applicable coordinate change restores performance. The same accessibility–reuse ordering appears in prospectively specified CIFAR-100 experiments with both ResNet-18 and DenseNet-121. Finally, preserving a candidate state that contains the distinctions needed by the anticipated task family improves reuse on compatible held-out tasks, with the clearest benefit when target supervision is limited. Representation reuse is therefore relative to the tasks that follow source learning: a useful representation must retain the distinctions those tasks need and make them available in a form that a shared downstream rule can use.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Reuse, Representation Learning, Structural Accessibility, Coordinate Consistency, Structure Preservation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WM5mOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks when a pretrained encoder stays useful for new tasks once its original head is removed. It argues that this depends on which future tasks one has in mind, not on how well the encoder solved its source task.

The theory (Sec. 2, App. A) says reuse can fail in two ways: the encoder has merged inputs that a new task needs to tell apart, or the information is there but encoded differently from input to input, so one decoder cannot read it (Eqs. 13–14). A task can be decoded exactly when neither happens (Thm. 1). The proposed method, SPR, adds an auxiliary loss that trains the encoder to predict a candidate task-relevant state alongside the source task (Eqs. 20–21).

Experiments on Shapes3D and CIFAR-100 (Sec. 3) show three things. Models with the same source accuracy differ in how well they support new tasks (Sec. 3.1). SPR raises low-shot reuse from 0.65 to 0.84, while random or shuffled auxiliary targets do not (Sec. 3.2). And a different invertible transformation applied to each sample's representation collapses reuse, from 0.41 to 0.02, until it is undone (Sec. 3.2). The SPR gain on held-out tasks is +0.17 in the low-shot regime and not resolved with full target supervision (Sec. 3.3).

Critical strengths

1. The formal statements are correct and carefully scoped. The conditions for exact reuse are stated precisely (Thm. 1, Eqs. 13–14), and the paper is explicit about what they do not imply: App. A.10 gives an example in which better state recovery yields worse reuse under a restricted decoder, and the text repeatedly limits the empirical claims to the tested families (L207–208, L416–418). This matters to my assessment because the formal part does not overclaim.

2. Careful experimental protocol. Comparisons are made between source-matched models, inference is at the level of independently trained seeds, task admissibility is decided without reference to the learned representations (App. B.5), and there are informative controls: random and shuffled auxiliary targets fail, while a global relabelling of the state labels behaves like SPR, consistent with insensitivity to the naming of state symbols (L374–375, L460–461). The accessibility trend is repeated on a second architecture. The reported numbers are internally consistent.

Critical weaknesses

1. The theory mostly restates its definitions, so it adds little beyond the framing. The framing itself is clear and useful, but I could not find a formal result that goes beyond what follows directly from the definitions, or one that predicts something measured in Sec. 3.

  • The task-relevant state is the tuple of task outputs. By Eqs. 5–6, two inputs are equivalent exactly when all task labels agree, so sH⋆s^\star_{\mathcal H} is, up to relabelling, (h1(x),…,hM(x))(h_1(x),\dots,h_M(x)). Lemma 1, Cor. 1, Cor. 2 and Prop. 2 then follow immediately.
  • Theorem 1 is the standard factorisation condition. It says a decoder exists if and only if the target is constant wherever interfaces coincide, which is the usual condition for one map to factor through another.
  • The remaining results are routine. Prop. 1 is a change of variables, Prop. 3 a triangle inequality with a union bound, and App. A.9 a Hoeffding bound.
  • The theory and the experiments are not connected. The paper says itself that the graded diagnostics used in Sec. 3 are not the exact conditions of the theory (L206–208, L367–368), that no monotone relationship between accessibility and reuse follows (L207–208, L417–418), and that the finite-sample bound does not predict the effect reported in Sec. 3.3 (L970–971).

This matters to my assessment because the theory is presented as the main contribution. It would be much stronger with one result that is not immediate: for example, a condition on source training under which a candidate state is provably preserved, or a bound that predicts the reuse gaps measured in Sec. 3.

2. The two main experimental effects are built into the setup. The experiments that are meant to support the theory's two failure modes illustrate the definitions rather than test them.

  • Coordinate consistency. The failure is created by applying a different invertible map to each sample and withholding which one (App. B.7). That a shared decoder then fails, and that the exact inverse restores the original value, holds by construction (L1229–1233). The paper gives no evidence that such sample-dependent conventions arise in trained encoders, yet the abstract presents this as an empirical finding (L023–026).
  • Benefit of SPR. The held-out tasks in Sec. 3.3 are functions of the candidate state ss, and SPR is trained with labels of ss. That it then supports few-shot tasks defined on ss better is expected, and the paper itself calls this the constructive case (L1416–1419). On novel rules that are not built from ss, SPR is slightly worse than the source-only model (Sec. 3.1).

A convincing test would show a consistency failure that arises from training, and an SPR gain on tasks that are not derived from the supervised state.

3. The experimental setup cannot be reconstructed from the paper. After reading the main text and App. B–D I could not determine:

  • the Shapes3D source task and encoder (App. D.2–D.3 describe only the CIFAR-100 encoders and refer to "the same relational source objective"), and what MATCH / MISMATCH / UNKNOWN means for a single image (App. B.2);
  • what the novel rules are (App. B.4) and what the coordinate transformations are (App. B.7, D.6);
  • the downstream decoder class and how it is fitted (App. D.4; a ridge readout is specified only for the oracle admissibility audit, Eq. 40), and the value of λextreuse\lambda_{ ext{reuse}} (Eq. 21);
  • the held-out task, chance level, seeds and intervals for the source-matched CIFAR-100 results (L356–360, App. C.5).

Without these the reported numbers cannot be interpreted or checked, and the release does not fill the gap: it has no source-training code and nothing for Secs. 3.1–3.2 or CIFAR-100.

3. The authors' own code release does not reproduce the headline result of Sec. 3.3. The release covers only this experiment, and its README says it is "not an exact numerical reproduction of manuscript Section 3.3" but an additional evaluation with a different readout. Under that evaluation the SPR gain is about 0.04 both with few and with many target examples, so there is no low-shot-specific advantage. The paper reports +0.17 against +0.02 (Table 4). The release states that its results "do not support a supervision-localized improvement claim", which is the claim in the title of Sec. 3.3 and in the introduction (L077–080). The paper does not mention this evaluation and does not say which readout produced its own numbers (App. D.4 says only that the readout class is fixed, L1862–1863). This matters because the one result I could check against the authors' code does not hold there.

Questions for authors

Clarifications

  1. Which theoretical statement is not an immediate consequence of the definitions in Eqs. 5–6 and 10–11, and which measured quantity in Sec. 3 does the theory predict?
  2. Please define the Shapes3D setup concretely: the source task and its labels, the per-image candidate state, the novel-rule registry, the encoder, the downstream decoder class and fitting procedure, and λreuse\lambda_{\text{reuse}}. Likewise the held-out tasks, chance levels, seeds and intervals for all CIFAR-100 results.
  3. What are the coordinate transformations in App. B.7, and is there any setting in which sample-dependent conventions arise from training rather than from an applied intervention?
  4. Fig. 4a appears to show absolute reuse falling as the number of target examples per state grows from 1 to 16, for all three conditions. Why does more target supervision lower reuse, and could the concentration of the SPR gain at small budgets be a consequence of this rather than of limited supervision as such?
  5. Which downstream readout, on which input, produced the Sec. 3.3 numbers, and why do they differ from the released evaluation, where the C1−C0 gain is about 0.04 in both regimes and Γ\Gamma is indistinguishable from zero? The release's output records a flag stating that a "historical semantic head path" is not consumed. Did the manuscript evaluation use the output of a head trained during source learning, which L1133 and L1851–1852 say is never supplied to the evaluator?

Minor fixes

The text is carefully written; I found no spelling or grammar errors. A few small things:

  • Terminology. The figures say "local-state" (Fig. 1, axis labels of Fig. 3) where the text says "candidate state", and "gauge" (Fig. 7 labels, "gauge-aware rescue" in App. C.3) where the text says "convention".
  • Undefined labels. C2 and C3 appear in the caption of Fig. 3 (L390), but the main text introduces only C0, C1 and C4 (L320–321); they are defined in Table 1 of App. B.1.
  • Dangling colons. Three sentences end in a colon while the table they introduce floats elsewhere (L1182, L1287, L1981).
  • References. "THEOFANIS KARALETSOS" is in capitals (L548). "ica" (L520), "imagenet" (L534) and "Leep" (L559) have lost their capitalisation.
  • Figure style. Fig. 4 uses a different font and colour scheme from the other figures.
4 comments
Reviewer_WM5mOpenReview ↗

Reply to the authors, Part 1/3: what the revision resolves, and the Shapes3D result

Thank you for the detailed responses and the revised manuscript. I have read all replies, including those to the other reviewers, the revised PDF and the updated supplementary release. Line numbers refer to the revised PDF.

What the revision resolves

  • Sec. 3.3 protocol. Thank you for correcting the statement that no source-trained head was used, and for withdrawing the low-shot-specific claim from the abstract (L031–032), Sec. 3.3 (L453–455) and the Conclusion. The strict head-free result (+0.0398 and +0.0424, with Γ=−0.0026\Gamma=-0.0026) is now the main evidence for encoder reuse. This addresses the central part of my original Weakness 3.
  • Scope of the claims. The paper now says that the coordinate intervention does not show that such conventions arise in training (L421–423, App. B.7). It also says that SPR's gains do not show an advantage over other objectives that use the same labels (L306–312). I agree with both statements.
  • Shapes3D setup. App. D.2–D.3 and B.4 now define the source task, encoder, auxiliary targets, training schedule and novel-rule registry.
  • Fig. 4a. The leave-one-family-out explanation in App. C.4 (L1865–1877) answers my question.

A note on soundness. My concerns were never about arithmetic errors. They are about whether the conclusions follow from the experiments as described. The points below are the ones that still decide my assessment.

1. The Shapes3D novel rules are built from the state that SPR is trained on

Correction to my original review. In my Weakness 2 I wrote that "on novel rules that are not built from ss, SPR is slightly worse than the source-only model". That statement was wrong. I repeated the paper's own description of these rules without checking it, and the original submission did not say what C1 was trained to predict in Shapes3D. With the details now given in App. D.3, the novel rules are built from that target, as shown below. I withdraw that part of Weakness 2. The correction does not help the paper's argument, though. It turns the Shapes3D result into evidence against the claim that SPR helps on compatible tasks.

In Shapes3D, C1 trains heads that predict the hue, scale and shape of each image (App. D.3, L2194–2200). Its candidate state is therefore s(x)=(hue,scale,shape)s(x)=(\text{hue},\text{scale},\text{shape}). Every novel rule is a Boolean function of whether these same three attributes agree between the two images (App. B.4, L1296–1307). So every novel rule has the form ϕ(s(x1),s(x2))\phi(s(x_1),s(x_2)) and belongs to HS(2)\mathcal H_S^{(2)} (Eq. 17). By Eq. 18, C1's training target contains every distinction the whole registry needs.

Yet C1 is worse than C0 by 0.0370 on all five seeds, with identical source accuracy (L379–381). The paper explains this by saying the registry does not require targets to "factor through the candidate state" (L392–393). The reply to Reviewer ZNcv says the same about "the candidate state preserved by SPR". Under the paper's own definitions this is not the case.

This matters because it is the only experiment with exactly matched source accuracy whose training and task registry are fully described. It is exactly the situation Sec. 2.4 is designed for, and SPR makes reuse worse there. A possible explanation is a restricted decoder, as in App. A.10. However, the readout used for the Shapes3D novel rules is not described anywhere (App. D.4 covers only the composition tasks), so this cannot be checked.

Reviewer_WM5mOpenReview ↗

Reply to the authors, Part 2/3: the released caches, and the missing setup

2. What the released caches show about the semantic-interface comparison

I looked at the released prediction caches together with the manifests.

  • C0's semantic predictor does not track the states. It agrees with the true local states in about 25% of cases, which is below the chance level of 1/3, while being confident (mean top probability about 0.91). Even after the best relabelling of its three outputs, agreement is only about 0.60 (0.47 to 0.67 across seeds). C1's predictor reaches about 0.95. The historical gain of +0.1715 therefore compares C1's trained state head with a C0 head that does not work, rather than comparing the two encoders. C0 has no auxiliary loss (L364). How was its predictor obtained?
  • Every held-out test set has a single label. Each held-out manifest contains 250 pairs from one composition family only, and the rule label depends only on the family. So within each evaluation cell all test pairs have the same target, and "reuse accuracy" for a cell is the share of pairs given that one label. App. B.2 says the target "may be constant" (L1233–1235). In the released data it is constant in every cell. Please state this in Sec. 3.3 and say which chance level applies.
  • Which readout produced the Sec. 3.2 recoverability numbers? C0's aligned macro-F1 in Sec. 3.2 (0.5547, L409–410) is close to what this source-trained head gives. App. D.5, however, describes a readout fitted to the frozen representation. A readout fitted to state labels could not score below chance before relabelling. If the numbers come from the source-trained head, the +0.41 gain shows whether a state head was trained, not what the encoder retains.

3. The setup behind the main SPR results is still missing

App. D.1 says the main controlled experiments use Shapes3D. But App. D.3 says the Shapes3D source task is different from the three-valued candidate states (L2191–2193). The Shapes3D encoder is also 256-dimensional (L2149), while the direct-z and coordinate experiments use 128-dimensional features (L416–417, L2235). The paper does not give:

  • the dataset, source task, encoder and λreuse\lambda_{\text{reuse}} of the MATCH/MISMATCH/UNKNOWN experiment behind Secs. 3.2 and 3.3,
  • what these three states mean for a single observation,
  • the task and number of classes in the coordinate experiment.

The release manifests point to an unnamed pool of data and do not answer these questions either.

Similar gaps remain elsewhere.

  • CIFAR-100 extensions (+0.1718 in the Introduction, +0.0768): what C1 is trained to predict, which readout is used, whether the evaluation is head-free, and seed-level intervals.
  • Q1–Q5 experiment: the source task and the held-out future tasks. If those tasks depend on the fine label, rising reuse from Q1 to Q5 is expected, because Aq=mq(C,F)A_q=m_q(C,F) supplies more fine-label information at each level. This is the supervision effect acknowledged at L306–312.
  • SAR experiment: the source task, encoder, candidate state, the six relation tasks, and how C2 and C3 are defined. There is one source seed and no intervals. The controls come close to C1 (72.68 and 74.75 against 76.09 at K=8K=8), C2 matches or beats C1 at K=40K=40, and the examples were chosen after the fact. In its current form it cannot carry the weight it has in Fig. 2.
Reviewer_WM5mOpenReview ↗

Reply to the authors, Part 3/3: theory, questions and minor points

4. Theory

The new construction in App. A.1 states explicitly that sH⋆s^\star_{\mathcal H} is the tuple of task outputs (h1(x),…,hM(x))(h_1(x),\dots,h_M(x)). This was the point of my original Weakness 1. The added results, the Bayes-error bound and the sharpness example for Prop. 3 with a constant decoder, are standard. The paper also confirms that the theory does not predict the measured effects (L230–232, L1078–1079). My view of the theoretical contribution is therefore unchanged.

Questions

  1. Are the Shapes3D novel rules in HS(2)\mathcal H_S^{(2)} for s=(hue,scale,shape)s=(\text{hue},\text{scale},\text{shape})? If so, how does C1 doing worse than C0 on all five seeds fit with Sec. 2.4 and the abstract? Which readout is used for the Shapes3D novel rules?
  2. How was C0's semantic-state predictor trained? Is the Sec. 3.2 recoverability diagnostic a readout fitted to the representation, or this source-trained head?
  3. Which dataset, source task and encoder produce the 128-dimensional representations used in Secs. 3.2 and 3.3 and in the coordinate experiment? What do MATCH, MISMATCH and UNKNOWN mean for a single observation?
  4. What does C1 predict in each CIFAR-100 extension, and what are the held-out tasks in the Q1–Q5 experiment?
  5. Can the SAR setup be described in full, and are results from more source seeds available?

Minor points

The reply lists the following as fixed, but they remain in the revised PDF:

  • "gauge" in Fig. 6 and App. C.3 (L1811), where the text says "convention",
  • sentences ending in a colon before a table that is placed elsewhere (L1387, L1509),
  • the reference entries "THEOFANIS KARALETSOS" (L602), "nonlinear ica" (L574), "imagenet" (L588) and "Leep" (L613).

In light of the above, I maintain my current assessment.

AuthorsOpenReview ↗

Final Response to Reviewer WM5m — Clarification of Outstanding Concerns

We note that the reviewer has not corrected the misunderstanding in Question 5 of the original review, which conflated results from two distinct evaluation protocols. Our audit confirmed no numerical inconsistency within the examined protocols. We consider this an error in the original review.

We also find the newly raised concerns difficult to understand, as several appear to address claims not made in our manuscript. For example, a negative Shapes3D result under a particular readout does not invalidate our theoretical characterization of representation sufficiency. This makes it difficult to identify what further revisions are being requested.

The reviewer acknowledges that the central concern in Weakness 3 has been resolved and explicitly withdraws an incorrect statement in Weakness 2, yet maintains the original minimum scores and rejection recommendation.

Given that we genuinely struggle to understand what the reviewer is arguing, we will not pursue the remaining points further and respectfully leave the assessment to the Area Chair.

Reviewer_4joiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work studies the reuse of learned representations and argues that reuse should be defined with respect to a family of future tasks rather than characterized by a single task-independent score. The paper defines the coarsest deterministic state sufficient for every task in a finite family and provides a criterion for exact reuse. It identifies two failure modes based on whether the required distinctions are recoverable from the representation and whether those distinctions are consistently interpretable to same downstream rule across samples. It further introduce SPR, a learning method targeting structural accessibility, and provides sufficient conditions for exact reuse based on candidate state recovery, with supporting empirical evidence.

Critical strengths
  • This paper introduces an insightful perspective on representation reuse, arguing that is should be defined with the respect to a family of future tasks. It provides a criterion for exact reuse and identifies two failure modes.
  • The main claim is supported by theoretical analysis empirical evidences.
Critical weaknesses
  • W1: The paper provides limited motivation or evidence for defining reuse with respect to a specified family of future tasks rather than characterizing it with a single task-independent score. Although Section 3.1 discusses `whether source-task success can rank learned representations for future reuse', this does not fully support their motivation.
  • W2: The justification and evaluation of SPR require clarification. (1) Proposed learning method SPR is motivated by the auxiliary objectives in multitask learning. How using SPR objective based on candidate states can be justified by provided motivation or references is not precisely explained. (2) Multitask learning and representation reuse have difference objective and settings. Since training SPR uses labels containing informations relevant to the anticipated future-task family, further in-depth analysis is required to investigate whether its gains come from its design or from additional task-relevant supervision. Random or shuffled auxiliary labels serve as ablation baseline, but do not fully address this concern. This limits the evidence for SPR’s methodological contribution.

That said, I am not fully familiar with this topic and am willing to revise my rating based on the discussion.

Questions for authors
  • Q1: I could not find an explanation of what SPR stands for in the main paper. Please correct me if I have overlooked it.
  • Q2: Some parts of the presentation are difficult to follow. Terms or settings are used before they are defined or explained in detail, requiring readers to look ahead for clarification. Also, providing (at least) a brief explanation of the experimental settings or how SPR is trained for the reported results in the main paper, rather than relying on the appendix, would help readers understand the claims.
  • Q3: How do the “matched representation pairs” work?
  • Q4: What reversible transformations for the effective interface are used in the experiments?
Reviewer_ZNcvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a task-relative theory of representation reuse, whose central claim is that source-task success does not equal reusability. The authors decompose reuse failure into two mechanisms, structural accessibility and coordinate consistency and formalize the core around the coarsest deterministic state sufficient for a specified future-task family, yielding a refinement lemma, an insufficient-state impossibility corollary, an effective-interface reuse criterion, family reuse as task-state recovery, and a restricted-reuse sandwich bound. The experiments validate the performance of the proposed method.

Critical strengths
  1. The conceptual split has genuine diagnostic value.
  2. The theory–experiment correspondence describes well.
  3. The selection of statistical inference unit shows the better performance.
Critical weaknesses
  1. For the core theoretical object, how to obtain one approximate and the authors do not provide construction procedure from task description to state partition.
  2. Do naturally trained representations exhibit endogenous, non-injected sample-dependent coordinate inconsistency? The paper provides no evidence of naturally occurring coordinate conflict.
  3. The sandwich bound in Proposition 3 is extremely loose in practice, but the paper uses it as theoretical support.
  4. Sections 3.1 and 3.3 point in opposite empirical directions, and the paper waves this away with different task families without any mechanistic test.
  5. The Q1–Q5 accessibility coordinate a_q is a property of the auxiliary target, not of the representation; the paper acknowledges this but slides toward the latter in its reading of Figure 10.
Questions for authors
  1. The two-architecture replication provides only all slopes are positive, without test of whether effect size depends on architecture.
  2. The interpretation of the random-auxiliary (C2) and shuffled-state (C3) controls is not unique.
  3. The coordinate intervention is performed on only one frozen representation, and the three source-training seeds merely repeat three versions of the same representation, so inter-representation heterogeneity of the intervention effect is untested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.