acceptodds
Under review as a conference paper at ICLR 2027

Shared-Factor Distribution Shift in Multi-Task Learning: Identification and Adaptation

Abstract

Multi-task models serve tasks whose labels arrive at different speeds. Clicks arrive within minutes and purchases only after weeks. After a shift, the slow tasks must be predicted from the fast tasks' early labels. Fine-tuning on those labels improves every task that can be checked, and worsens a slow one. Which part of a slow task's shift do the early labels determine, and what should be done about the rest? We formalize the shift as window-specific gains on a few factors shared by all tasks, and derive what the early labels determine. A slow task's shift is identified if and only if every factor it reads is read by some fast task, and two past windows identify the factors themselves. Because the part that no fast task reads is unknown, an update that reaches into it can lose by any amount; the best update that avoids it is the refit with its unread part removed. FLAG serves this projected update on a frozen multi-task model. Replayed on four datasets and 47 weeks of a question-answering site, it has the lowest slow-task error of eight baselines, and is worse than not adapting in none of 89 served windows.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multi-task learning, distribution shift, task-environment interactions, identifiability, partial identification, delayed feedback

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ybeHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies adaptation under distribution shift when fast-task labels arrive before slow-task labels. Assuming shared factors with window-specific gains, it characterizes which slow-task changes are identifiable from early labels, explains why ordinary fine-tuning can harm unobserved tasks, and derives a projected update that avoids dependence on unidentified changes. FLAG implements this principle on frozen multi-task models and is evaluated through simulations and temporal replays

Critical strengths
  1. The paper identifies a meaningful failure mode beyond conventional gradient conflict. Improving every observable fast task can still harm an unobserved slow task because the induced fast-to-slow relationship is inappropriate after the shift. The distinction between factor coverage and readout span makes this explanation precise and provides a useful conceptual contribution.

  2. The method follows naturally from the identification analysis. The paper distinguishes changes that early labels reveal from those they cannot determine, then derives a task-specific projection rather than filling unidentified directions from potentially stale correlations. The decision-theoretic justification gives a clear reason for the update’s structure.

  3. The experiments directly test several theoretical predictions. Projection ablations, comparisons with filling unread directions, per-task results, coverage diagnostics, and assumption-violation tests go beyond routine benchmarking. The temporal replays also connect the theory to realistic differences in label availability.

Critical weaknesses
  1. The safety claims require more precise framing. Theorem 1 establishes nonnegative improvement averaged over a specified family of shift distributions, not protection in every individual window. Similarly, the empirical “never worse” count applies a standard-error threshold, while the appendix reports losses in individual task-window pairs. These distinctions should remain explicit in the abstract and conclusions.

  2. The stable-factor/readout assumption materially limits deployment guarantees. Appendix G.3 appropriately tests violations and shows that slow-task readout drift can substantially increase harmful updates. Such changes are invisible to fast labels, and a check using matured windows cannot immediately detect a newly occurring violation. This is an important limitation of the proposed safety interpretation.

  3. The practical procedure introduces estimation and approximation beyond the central theorem. Learned factors, estimated information matrices, thresholded weak directions, bootstrap estimates of shift size, and binary-task linearization all affect the update. The appendices analyze several pieces, but the full implementation does not automatically inherit the population guarantee without additional conditions.

Questions for authors
  1. Can the claims consistently distinguish average-risk safety, statistically detectable degradation, and individual-window harm?

  2. Which guarantees hold for the complete estimated implementation, and which should be interpreted as guidance from the idealized model?

  3. Under readout drift, how much protection does the actual matured-window fallback provide, including its detection delay?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ybeH

Many thanks for three questions that the revised paper can answer sentence by sentence.

Q1. The three notions now have three separate wordings

"Can the claims consistently distinguish average-risk safety, statistically detectable degradation, and individual-window harm?"

Yes. The abstract, Section 5 and the conclusion now state the theory's guarantee, the replay result and single-task losses in three separate sentences, quoted in the table.

The theory also explains why the three must stay apart. If a shift in the directions the fast tasks read could take any value, a window with no shift at all would be possible, and any update that moves the model would lose there. Only not adapting can promise never to lose in a single window (Proposition 10(ii)).

NotionRevised textWhere
Average-risk safety"The projected update's improvement is the same whatever the shift does in directions no fast task reads, and non-negative on average over shifts in the directions they read, at the size seen in past windows. It does not protect every window."Section 3.4
Detectable degradation"…and in none of the 89 served windows is its slow-task mean detectably worse than not adapting."Abstract; Sections 1, 5 and 7
Individual-window harm"Single slow tasks lose to not adapting in 11 of the 114 (window, slow task) pairs of the four shorter replays, and upvotes in 9 of the 47 weekly windows, each within the label noise."Section 5.2 and the conclusion

Q2. The implemented update inherits the projection exactly and relies on the learned factors for the rest

"Which guarantees hold for the complete estimated implementation, and which should be interpreted as guidance from the idealized model?"

A new paragraph closing Section 4 sorts the results into three tiers: what the served update inherits exactly, what it keeps approximately, and what depends on the learned factors being right.

Lemma 1 and the risk results use only the served model's features, not the factor form behind them (Appendix E).

  • Inherited exactly, on the linear model of Section 3.1. The update's improvement does not depend on the shift along the directions FLAG treats as unread, weakly read ones included (Proposition 8). In all 114 pairs, using the weak information there could have raised the worst case by under 1% of the predicted gain, and the condition for a larger true shift to keep the guarantee held (Propositions 9 and 10; Appendix H.4).
  • Kept approximately. Uncertain estimates lean toward not adapting: the typical shift size is a low bootstrap quantile, and the label information is capped at its model-based value. Binary tasks and the window-embedding gate use a model linearized at a pilot fit (Appendices E.1 and E.2).
  • Relies on the learned factors. Theorem 1 takes the learned factors and readouts as given, so their error acts as model error that no early label reveals (Appendix C.1). On matured windows, the part of the error the model cannot represent is under a tenth of the gap between the baselines and FLAG (Section 5.2).

Q3. The check protects once the drifted labels mature, and their maturity sets its delay

"Under readout drift, how much protection does the actual matured-window fallback provide, including its detection delay?"

Protection starts when a drifted window's slow labels mature. The check then switches a harmed task back to the trained model after one to three windows in the four shorter replays, and mostly leaves a still-helpful update on.

The protection has two layers. A shift along directions no fast task reads cannot alter the projected update's improvement, even in its own window (Theorem 1). A change to the slow task's own readout shows only in that task's labels, so no test on the fast labels can beat chance in the worst case (Proposition 6). The check covers this second part. It reads the latest matured window: 1 window back on KuaiRand and G2F, 2 on CrossValidated and 3 on RetailRocket (Appendix H.2).

To measure it, we ran the random-task setting of Appendix G.3 as 1,000 sequences of 20 windows, a deliberate violation of (A0). A readout drift starts in window 6, and the check runs at lag 1 or 3. Each cell is the share of drifted windows where the served update loses to not adapting.

Readout driftNo checkCheck, lag 1Check, lag 3
60% of the change's variance (update still helps on average)33%26%28%
The slow task's weights on the factors it reads flip sign (update harmful on average)78%9%21%

Under the harmful drift, the median delay before switching back is 1 drifted window at lag 1 and 3 at lag 3. Under the mild drift, the check keeps 93% of the update's mean gain. Sections 4 and 7 will present the check as a guard against persistent misspecification, apart from the safety guarantee.

Dear Reviewer ybeH,

Thank you again for your questions on the safety claims. Before the discussion period closes, here are our answers in brief.

  • Q1. Average-risk safety is what Theorem 1 guarantees: the improvement is non-negative on average, not in every window. Detectable degradation occurs in none of the 89 replay windows. Individual-window harm does occur, in 11 of the 114 (window, slow task) pairs, within label noise. The abstract, Section 5 and the conclusion now state these three separately.
  • Q2. On the linear model of Section 3.1, the implemented update keeps one property exactly: its improvement does not depend on shifts that no fast task's labels inform (Proposition 8). Uncertain estimates lean toward keeping the trained model, and the remaining guarantee relies on the learned factors being accurate (Appendix C.1). A new paragraph closing Section 4 separates these three parts.
  • Q3. The fallback compares FLAG with the trained model on the latest window whose slow labels have arrived. In a new simulation where the slow task's own weights flip sign, checking one window back cuts the share of losing drifted windows from 78% to 9%. Checking three windows back cuts it to 21%. The median delay before switching back is one and three windows, respectively.

Please refer to our detailed response above for the full results. Does this match the distinctions you had in mind?

Best regards,

The Authors

Thanks a lot for the authors' detailed rebuttal. I'll keep my original positive rating.

Reviewer_m8xbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method (FLAG) for adapting frozen multi-task models to distribution shifts when labels for some tasks arrive earlier than others. It shows that fine-tuning on fast labels can silently hurt slow tasks, because it implicitly applies a surrogate relation fixed at pretraining. FLAG projects out update directions that cannot be identified from the available labels, aiming to avoid harmful transfer of knowledge.

Experiments on four datasets and a 47-week CrossValidated replay show that FLAG outperforms existing baselines in slow-task prediction error and is never significantly worse than not adapting across 89 evaluated windows. Its theoretical guarantees rely on restrictive assumptions about stable task readouts and shared-factor shifts.

Critical strengths
  • The paper formalizes an important problem, explains why standard fine-tuning can harm slow tasks, and derives a principled update with a clear safety objective.
  • Well-written paper.
  • The experimental design is sound. It evaluates multiple datasets, compares against diverse baselines, and includes ablations that support the importance of the projection step.
Critical weaknesses
  • The method assumes that distribution shifts can be represented by changes in shared-factor gains while the factors and task readouts remain stable. The paper acknowledges that task-specific changes violate this assumption, which limits generalizability.

  • Although the experiments span four datasets, the reported results do not establish how robust FLAG is when shared factors or task readouts themselves change.

  • The fallback mechanism also contributes to the empirical safety result, so the "never worse" finding should not be interpreted as a universal guarantee. Also, what is the cost-accuracy tradeoff?

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer m8xb

We are grateful for questions that read the results as a practitioner would, asking what the headline rests on and what it costs.

W3 (part 1). The safety result holds without the fallback, and its wording is now exact

"The fallback mechanism also contributes to the empirical safety result, so the "never worse" finding should not be interpreted as a universal guarantee."

With the fallback switched off, FLAG is still not detectably worse than not adapting in any of the 89 replay windows. The revised text states it as an empirical count, not a guarantee.

The fallback switches a slow task back to the trained model when FLAG lacks enough past windows or when the check on the latest matured window finds FLAG worse there. It fired once, for the three slow tasks of G2F's first served year, where FLAG did not yet have enough past windows (Appendix H.2). In the other 111 of the 114 (window, slow task) pairs of the four shorter replays (all but the weekly one), FLAG served its own update (Table 20). Section 5.2 reports the run without the fallback.

The claim now reads: "in none of the 89 served windows is its slow-task mean detectably worse than not adapting." Detectably worse means higher by more than the two methods' summed standard errors over three seeds. Single slow tasks still lose in 11 of the 114 pairs, and upvotes in 9 of the 47 weekly windows, each within the label noise.

W1 and W2. FLAG was tested when readouts change, on real data and in deliberate violations

"…the reported results do not establish how robust FLAG is when shared factors or task readouts themselves change."

On real data, rescaling the shared factors captures most of each slow task's change. When a slow task's readout drifts in simulation, FLAG's update still beats not adapting on average and loses less often than fine-tuning.

(A0) fixes each task's weights on the shared factors, its readout, and lets each window rescale the factors, which already changes how tasks correlate. Between consecutive matured windows, refitting only the rescaling removes 81 to 93% of the slow tasks' error that a free refit of all weights removes. Fitted without one slow task's labels, it still removes 71 to 84% of what a free refit removes on that task, against 38 to 55% for an adapter with as many parameters (Appendix H.4). The replays in Table 1 therefore already run where (A0) holds only approximately.

Appendix G.3 breaks (A0) on purpose in 6,000 simulated slow-task windows. A loss counts any excess error over not adapting, with no threshold.

Violation of (A0)Its share of the slow task's changeFLAG's update losesFine-tuning losesFLAG's mean error vs not adapting
The slow task's readout drifts10 / 30 / 60%10 / 23 / 33% of windows29 / 35 / 40%31 to 73% lower
A new factor only the slow task reads10 / 30 / 60%3.2 / 9.2 / 24%26 / 28 / 34%33 to 74% lower

Both violations show up only in the slow task's own labels (Proposition 6). The check on matured windows reacts once those labels arrive (reply to ybeH). The revision moves these tests into Section 5.

W3 (part 2). Compute is negligible, and the price of safety is a declined bet on past patterns

"Also, what is the cost-accuracy tradeoff?"

FLAG adds 0.4 to 2.1 CPU seconds per window. Its real price is the gain it forgoes when past co-movement repeats in directions no fast task reads, and on real data betting on that repetition loses on average.

This is the trade that minimax-regret rules make against a status quo (Manski, 2004): a smaller gain when the past repeats, in return for an improvement that holds whether or not it does. The early labels cannot tell whether the past repeats, so no rule computed from them can pick the better choice (Corollary 3). In the replays, extending past co-movement in every window has higher mean error than FLAG on every dataset and is detectably worse in 15 windows (Table 1).

A toy example from Section 3.4 shows the trade. Two campaigns change appeal and quality alike, and purchase intent either follows the past co-movement or breaks from it. Improvements are in units of r2r^2, where rr is how far the campaign moves away from past windows.

Improvement over not adaptingIntent follows the pastIntent breaks from it
Extend past co-movement to intent+.294r2+.294r^2−.028r2-.028r^2
FLAG+.226r2+.226r^2+.226r2+.226r^2
  • When history allows. With at least 40 matured windows, FLAG can test the bet on them and take it only where it has paid (Appendix F.1). In the weekly replay this lowers error from .961 to .958, outside the guarantee (Table 2).
  • Partly read tasks. Slow tasks whose factors the fast tasks read only partly, such as purchases and upvotes, keep under 45% of the unprojected update. Well-read tasks keep over 90% (Section 5.2).

Dear Reviewer m8xb,

We appreciated your questions on the fallback, the fixed-readout assumption, and what safety costs. Before the discussion period closes, we would like to briefly recap our answers.

FLAG's safety result holds without the fallback: with it switched off, none of the 89 replay windows is detectably worse than not adapting (Section 5.2). The fallback, which switches a task back to the trained model, fired only once, in the first year of the G2F replay, because FLAG did not yet have enough past windows. The revised text now states the result as an empirical count rather than a guarantee.

On real data, rescaling the shared factors alone removes 81 to 93% of the slow tasks' error that refitting all weights removes (Appendix H.4). When we deliberately let a slow task's own weights drift in simulation, FLAG loses to not adapting in 10 to 33% of windows, against 29 to 40% for fine-tuning (Appendix G.3).

On cost, FLAG adds 0.4 to 2.1 CPU seconds per window. Its accuracy cost is that it does not bet on past correlations between tasks repeating, and on real data that bet loses on average (Table 1).

Please refer to our detailed response above for the full results. Do these settle your concerns about the fallback and robustness?

Best regards,

The Authors

Reviewer_m8xbOpenReview ↗

Response by Reviewer m8xb

Thank you to the authors for the explanations and manuscript revisions. I would like to keep my original score.

Reviewer_FbLaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies distribution shifts in multi-task learning when some tasks receive labels earlier than others. The authors analyze which parts of a slow task's distribution shift can be identified from the early labels of fast tasks. Based on this analysis, they propose FLAG, which estimates shared factor changes and projects out directions that cannot be reliably identified. Experiments on four datasets show that FLAG improves slow-task performance while reducing the risk of negative transfer.

Critical strengths

1.The paper addresses an interesting and practical problem. The motivation is clear, especially the observation that fine-tuning on fast tasks can hurt slow tasks even when the observed tasks improve.

2.The theoretical analysis is a strong part of the paper. The identification condition and the derivation of the projected update provide a useful explanation of when adaptation is safe.

3.The experiments are fairly comprehensive, covering different datasets and temporal settings. The ablation studies also provide good evidence that the projection step is important for avoiding negative transfer.

Critical weaknesses

1.The method relies on a strong assumption that distribution shifts mainly affect shared factor gains while the factors and task-specific readouts remain fixed. This may not hold in real applications, where task relationships can also change.

2.The evaluation is based on historical data replays. While this provides a controlled setting for comparison, it is unclear how well FLAG would work in real deployments where label delays and update schedules may be less predictable.

3.The baseline comparisons could be stronger. Since FLAG estimates time-varying shared factors, it would be useful to compare against dynamic factor or online state-estimation methods that also use early labels. The current comparisons make it somewhat difficult to separate the benefit of the factor model from that of the safety projection.

Questions for authors

please see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer FbLa

Thank you for naming the exact comparison you would trust. We ran it and answer your third point first, since its result frames the other two.

W3. The new dynamic-factor baseline shows that safety comes from the projection

"…compare against dynamic factor or online state-estimation methods that also use early labels… separate the benefit of the factor model from that of the safety projection."

We added the Kalman-filter dynamic factor model from nowcasting, with and without our projection. Unprojected, it is detectably worse than not adapting in 7 of 89 windows, and projected, in none.

FLAG estimates how a shift rescales a few shared factors, then projects out the directions that no fast task's labels inform. The new filter (Giannone et al., 2008; Bańbura & Modugno, 2014) estimates the same rescaling from the same early labels, with a random-walk or shrunken VAR(1) transition and the same 20-configuration tuning as every method (Appendix H.2), so only the estimator changes. "Detectably worse" means above not adapting by more than the summed standard errors over three seeds.

Slow-task error (lower is better)KuaiRandRetailRocketCV, quarterlyG2FCV, weeklyDetectably worse (of 89)
Not adapting1.024.829.966.879.971—
Refit of the rescaling, no projection.995.814.905.858.9644
Refit + past co-movement for unread directions1.007.822.917.870.96515
Filter, no projection (new)1.000.817.910.864.9667
Refit + projection (FLAG).978.801.885.842.9610
Filter + projection (new).982.803.890.846.9620

The refit alone gives half to three fifths of FLAG's mean gain over the best earlier baseline, and the projection gives the rest and the safety. With the projection, the two estimators differ by only .001 to .005. In filtering terms, our coverage condition is observability: along a direction no fast task reads, the observations carry no signal, and a filter moves there only through past correlations.

The unprojected filter becomes the ninth and strongest baseline. FLAG's error is below it by .005 to .025 (0.5 to 2.7%), with 95% paired bootstrap intervals excluding zero (Tables 1 and 18).

W1. (A0) lets task relationships change, and real data show how much it captures

"…the factors and task-specific readouts remain fixed. This may not hold in real applications, where task relationships can also change."

Each task weighs the shared factors differently, so task relationships change whenever a window rescales the factors. On real data, this rescaling explains most of each slow task's change.

We appreciate this point, since the rest of the paper leans on (A0), which fixes each task's weights on the shared factors. Figure 3a shows the correlations between tasks moving from window to window, which (A0) accommodates. A change in one task's own weights is the other kind. When it shows in the fast labels, an overidentification test can detect it (Sargan, 1958; Hansen, 1982). When it touches only the slow task, such as a new checkout step that alters purchases alone, only that task's own labels can reveal it (Proposition 6).

  • Real data (Appendix H.4). Refitting only the rescaling, without one slow task's labels, removes 71 to 84% of what a free refit removes on that task, against 38 to 55% for an adapter with as many parameters.
  • Deliberate violation (Appendix G.3). When a drift in the slow task's own weights carries 10, 30 or 60% of its change, the projected update loses to not adapting in 10, 23 and 33% of simulated windows, counting any excess error. Fine-tuning loses in 29, 35 and 40%, and its mean error stays 31 to 73% below not adapting's.

The revision moves both tests into Section 5.

W2. FLAG uses whatever labels have arrived, so irregular timing changes how much it adapts

"…it is unclear how well FLAG would work in real deployments where label delays and update schedules may be less predictable."

FLAG works with whichever labels have arrived, and the replays already run on uneven label timing, recorded on CrossValidated and reconstructed from event order elsewhere.

With no early labels, FLAG keeps the trained model, and each further label can only shrink the uninformed part, so its predicted gain never falls (Proposition 17).

  • Uneven arrival (Appendix H.1). At each served RetailRocket week, 190,000 to 250,000 past purchase labels are still immature and masked, and only 60 to 70% of early fast labels arrive by the cutoff.
  • Slow outcomes seen early (Section 5.3). By the weekly cutoff, 45% of acceptances and 77% of upvotes are seen. Taking them as slow labels, FLAG reaches .953 and is detectably worse in none of the 94 task-weeks. A delayed-feedback model reaches .962 and is worse in 8.

Dear Reviewer FbLa,

Thank you again for your review. Before the discussion period closes, we wanted to briefly check whether our response addressed your three points.

Dynamic-factor baseline (W3). The submission lacked this comparison. We added the Kalman-filter dynamic factor model you suggested, with and without our projection, the step that removes update directions no fast task's labels inform. Without the projection, the filter is detectably worse than not adapting (keeping the trained model) in 7 of 89 windows, and with it in none (Tables 1 and 18). Detectably worse means higher by more than the summed standard errors over three seeds. With the projection, the two models differ in mean error by only .001 to .005, so the safety comes from the projection.

Changing task relationships (W1). Our assumption fixes each task's weights on the shared factors, while each time window rescales the factors. Because tasks weigh the factors differently, this rescaling alone changes how tasks relate. Even without a slow task's own labels, it removes 71 to 84% of the error that refitting all weights removes on that task (Appendix H.4). When we deliberately let that task's own weights drift in simulation, FLAG still loses to not adapting less often than fine-tuning (Appendix G.3).

Less predictable label timing (W2). FLAG simply uses whichever labels have arrived. With none, it keeps the trained model, and more labels never lower its predicted gain (Proposition 17).

Please refer to our detailed response above for the full results. We would be glad to hear whether these address your concerns.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the three reviewers for asking the questions a real deployment would ask. As a reminder of the setting, FLAG updates a trained multi-task model after a distribution shift, using the early labels of fast tasks, such as clicks, to update slow tasks, such as purchases, before their own labels arrive. "Not adapting" means keeping the trained model as it is, and every result below is measured against it.

The reviews converge on three topics, and each section below opens with its answer.

1. The new baseline shows that safety comes from the projection

We added the Kalman-filter dynamic factor model that FbLa asked for. Without our projection it is detectably worse than not adapting in 7 of 89 windows, and with it in none.

FLAG first estimates how the shift moved a few shared factors, then removes from the update every direction that no fast task's labels inform. This removal is the projection. The new filter estimates the same factor rescaling from the same early labels, so comparing it with and without the projection separates the estimator from the projection (reply to FbLa). With the projection, the two estimators differ in mean error by only .001 to .005. The unprojected filter becomes the ninth and strongest baseline, and FLAG keeps the lowest error on all five replays.

2. Assumption (A0) is tested on real data and under deliberate violation

On real data, rescaling the shared factors explains most of each slow task's change. When we break (A0) on purpose, FLAG still beats not adapting on average and loses less often than fine-tuning.

(A0) says that each task keeps fixed weights on a few shared factors, while each time window rescales the factors. Because tasks weigh the factors differently, this rescaling alone changes how tasks correlate, as Figure 3a shows on real data.

  • Real data (Appendix H.4). Refitting only the factors' rescaling, without one slow task's labels, still removes 71 to 84% of the error that a free refit of all weights removes on that task, against 38 to 55% for an adapter with as many parameters. The replays in Table 1 therefore already run where (A0) holds only approximately.
  • Deliberate violation (Appendix G.3). When a drift in the slow task's own weights carries 10, 30 or 60% of its change, FLAG's update loses to not adapting in 10, 23 and 33% of simulated windows, against 29, 35 and 40% for fine-tuning. Its mean error stays 31 to 73% below not adapting's.

A change in one slow task's own weights shows up only in that task's labels (Proposition 6). A check on windows whose slow labels have matured catches it once those labels arrive, after a delay (reply to ybeH).

3. The safety claim is stated at three levels

We now state safety separately for the theory, for the replays and for single tasks, and the abstract, Section 5 and the conclusion use this wording.

LevelWhat holdsWhere
TheoryThe improvement is the same whatever the shift does in directions no fast task reads, and non-negative on average over shifts in the directions they read, at the size seen in past windows. It does not protect every windowTheorem 1
ReplaysIn none of the 89 served windows is the slow-task mean detectably worse than not adapting, that is, higher by more than the summed standard errors over three seedsTable 1
Single tasksA single slow task loses in 11 of the 114 (window, slow task) pairs of the four shorter replays, and upvotes lose in 9 of 47 weekly windows, each within the label noiseTable 20, Section 5.3

The last sentence of the abstract will read: "…and in none of the 89 served windows is its slow-task mean detectably worse than not adapting."

Changes to the paper

  1. The dynamic-factor baseline, the ninth, with and without our projection, on all five replays (Tables 1 and 18).
  2. The three-level safety wording above, in the abstract and Sections 1, 3.4, 5 and 7.
  3. The (A0) tests move from Appendix G.3 into Section 5, next to the real-data test.
  4. The matured-label check is described in Sections 4 and 7, with its delay and a new simulation (reply to ybeH).
  5. A paragraph closing Section 4 separates what the implemented method inherits from Theorem 1 and what it does not (reply to ybeH).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.