acceptodds
Withdrawn from ICLR 2027

FORTA: Feedback-Guided Predictive Distribution Repair under Heterogeneous Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time adaptation is difficult because the appropriate correction can change as target conditions evolve. Methods built around a particular adaptation mechanism restrict how the predictive distribution can be corrected. This work studies test-time adaptation with sparse delayed feedback: predictions are made before labels are available, and only a small fraction of labels are released later. Adaptation is formulated as learning and allocating a time-varying set of predictive distribution corrections from a shared frozen representation. FORTA maintains complementary source-anchored corrections. Every supervised member continues learning from the same released feedback regardless of its current weight, while saved pre-feedback predictions determine future influence. This separates how the correction set evolves from which corrections shape the next prediction. The analysis characterizes an information boundary for label-free conditional adaptation and, in a source-anchored construction, establishes an allocation advantage over separately tuned anchors and fixed mixtures of optimized candidates. Across seven image and signal benchmarks, FORTA improves macro-F1 over the source predictor. We verify how feedback availability and regime duration affect adaptation, and when adaptive allocation improves predictive performance, through synthetic and matched-portfolio experiments.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, online learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AvhuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FORTA, a test-time adaptation framework for heterogeneous streams with sparse delayed feedback. FORTA maintains a time-varying set of source-anchored predictive distribution corrections, trains all supervised candidates from released labels, and uses saved pre-feedback predictions to dynamically allocate their future influence. The paper provides an information-theoretic lower bound for conditional adaptation before informative feedback arrives, a source-anchored construction showing an allocation advantage over fixed mixtures, and experiments on seven image and signal benchmarks.

Critical strengths
  • The problem setting is important and realistic: predictions must precede label queries, only a small fraction of labels are observed, and feedback arrives after a delay.

  • Separating candidate learning from candidate allocation is a clear and useful design choice. Low-weight candidates continue learning and can regain influence when the target regime changes.

  • The use of saved pre-feedback predictions is methodologically appropriate, since a released label should affect only future predictions and should evaluate the prediction that was actually made.

  • The paper includes controlled switching and matched-portfolio experiments that attempt to isolate the contribution of adaptive allocation from the contribution of candidate learning.

  • The analysis explicitly characterizes the difficulty of identifying conditional changes from unlabeled inputs and discusses the roles of feedback coverage, delay, and regime duration.

  • FORTA improves the average macro-F1 over the source predictor and most reported comparators across the image and signal benchmarks.

Critical weaknesses
  • The central allocation mechanism appears closely related to online expert aggregation, Hedge, fixed-share, stacking, and delayed online model selection. Although these methods are discussed, the paper does not yet establish a sufficiently sharp distinction between FORTA and applying a standard delayed expert-tracking algorithm to the same candidate predictions.

  • The controlled allocation results do not show a consistent advantage over standard adaptive aggregation. In Figure 3, FORTA is only marginally better than uniform pooling on shuffled PACS and is worse than some fixed-share, Hedge, or convex-stacking comparators in several conditions. This makes it difficult to determine whether the proposed routing rule, rather than the existence of a candidate portfolio, is responsible for the reported gains.

  • The positive theoretical result is highly construction-specific. Theorem 5.2 relies on a source-anchored setting with two regimes, complementary corrections, a specified discounted router, sufficiently long segments, and empirical anchored minimizers. The result explicitly does not cover unrestricted zero-anchor adaptation, input-dependent anchor selection, jointly trained mixtures, arbitrary shifts, or the deployed neural candidate pools. The paper should make this limited scope more prominent and avoid presenting the theorem as a general guarantee for FORTA.

  • The feedback model is idealized. Queries are sampled uniformly and independently of confidence, class, or input difficulty, whereas practical feedback is often biased toward uncertain, high-risk, or manually selected examples. Since the paper emphasizes feedback coverage as a central factor, the method should be evaluated under non-uniform and class-imbalanced feedback.

  • The main signal results use only three runs. Several differences are small, for example FORTA versus Audit replay on WISDM and Gas, but no paired significance tests or confidence intervals are reported. Mean and standard deviation alone are insufficient to establish that these differences are reliable.

  • The paper states that the real benchmarks were used during development and are not an independent confirmatory suite. This substantially weakens the evidential value of the benchmark results, especially because the reported PACS reset result is not fully reproducible: the archived result is 0.794 whereas the rerun obtains 0.788, and the discrepancy remains unresolved.

  • The benchmark comparisons do not fully separate candidate learning from allocation in the natural image and signal settings. The matched-portfolio experiments are useful, but the paper should report absolute performance for delayed Hedge, fixed-share, and other adaptive aggregation baselines under exactly the same candidate trajectories, feedback identities, and initialization.

  • FORTA uses modality-specific routing rules and hyperparameters: class-specific Brier-based allocation for images and global macro-F1-based allocation for signals. This raises questions about how much of the reported performance comes from the general FORTA framework and how much comes from task-specific router design.

  • The computational and memory costs are not adequately characterized. Maintaining multiple correction mechanisms, replay buffers, delayed prediction records, and router statistics may be substantially more expensive than a single TTA method. The paper should report inference overhead, update cost, memory usage, and scaling with the number of candidates and replay size.

  • The recovery behavior is not always immediate. The paper acknowledges that the router can retain mass on a previously useful correction after the source model becomes optimal. This is important in rapidly changing streams, where delayed feedback and routing memory may make FORTA worse than the source predictor or a simpler fixed strategy.

  • The pseudo-oracle and synthetic results demonstrate that adaptive allocation can help in carefully controlled regimes, but they do not establish that the same advantage persists under realistic candidate misspecification, simultaneous covariate and conditional shift, noisy labels, or feedback that does not cover the changed region.

  • The paper reports strong average performance, but FORTA is not the best method on every dataset. In the signal experiments, Audit replay remains stronger on WISDM and Gas. The claims should therefore be framed around average robustness across heterogeneous conditions rather than universal superiority.

Questions for authors
  1. Under exactly the same candidate trajectories, feedback identities, delays, and initialization, can the authors report absolute performance and paired confidence intervals or significance tests for delayed Hedge, fixed-share, convex stacking, uniform pooling, and FORTA? This comparison would clarify whether the gain comes from the routing rule or from the candidate portfolio itself.

  2. How does FORTA perform under non-uniform, class-imbalanced, confidence-dependent, noisy, or coverage-limited feedback, and under simultaneous covariate and conditional shift? These settings would test whether the conclusions extend beyond the idealized feedback model.

  3. Can the authors clarify the scope of Theorem 5.2, resolve the PACS reset discrepancy (0.794 archived versus 0.788 rerun), and report inference overhead, update cost, memory usage, and scaling with the number of candidates and replay size?

Reviewer_FMc4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FORTA for test-time adaptation with sparse, delayed labels. A frozen source model supports a heterogeneous set of predictive-distribution corrections. Every supervised candidate learns from the same released labels independently of its mixture weight, while saved pre-feedback predictions determine future allocation. The analysis gives an information bound before informative feedback and an allocation advantage in a specified source-anchored construction with optimized candidates. Experiments cover seven image and signal benchmarks, controlled regime switching, feedback sensitivity, and matched-trajectory comparisons of combiners. The results support the complete framework and demonstrate that the benefit of its particular allocation rule depends on the stream and comparator

Critical strengths
  1. The paper proposes a coherent framework for heterogeneous corrections, making different adaptation mechanisms interoperable at the predictive-distribution level without modifying the source encoder.
  2. Theoretical analysis and experimental results show effectiveness of the method.
Critical weaknesses
  1. Novelty is limited. HILTTA (Li et al., TMLR 2024) already uses sparse human labels for model selection and supervised updating, with candidates arising from different hyperparameters. MORPHEUS (Danilowski et al., ICLR 2026 TTU Workshop) selects among TTA algorithms using an offline-trained predictor of adaptation performance. ReservoirTTA (NeurIPS 2025) and MoETTA (AAAI 2026) address expert specialization and routing. Their proximity makes the broad formulation of adapting multiple candidates and selecting their influence insufficient as a novelty argument. The paper should identify the nontrivial advance specifically attributable to heterogeneous corrections, weight-independent ongoing learning, and delayed saved-output evaluation.
  2. Incomplete attribution of key design choices. The matched-pool study does not directly test feedback-driven hard selection or input-conditioned gating on the same candidates. A HILTTA-inspired selector based on smoothed pre-feedback validation losses and a gate trained only on released labels would help distinguish the value of FORTA’s specific routing choice. These would be controlled adaptations, not reproductions of the original HILTTA or MoE systems. Moreover, class-specific image weights add capacity beyond the global controls, and Appendix F gives only qualitative class-offset ablation results. Finally, all matched combiners inherit the same candidate training, so Figure 3 cannot independently establish the benefit of continually training suppressed candidates. Focused controls would clarify the contributions of selection, combination capacity, and continued learning. Theorem 5.2 does not resolve these questions for the deployed heterogeneous bank.
  3. Organization and writing is poor and do not fit the conference standards.

References

Danilowski, M., Murphy, A., Kwon, Y. D., & Ghosh, A. (2026, April 27). MORPHEUS: Meta test-time adaptation via neural collapse geometry The 3rd Workshop on Test-Time Updates (TTU), International Conference on Learning Representations (ICLR) 2026.

Fan, X., Jiang, J., Chen, Z., Huang, F., Chen, X., Jiang, Q., Zhang, B., Tang, X., & Wang, Z. (2026). MoETTA: Test-time adaptation under mixed distribution shifts with MoE-LayerNorm. Proceedings of the AAAI Conference on Artificial Intelligence, 40(25), 21011–21019.

Li, Y., Su, Y., Yang, X., Jia, K., & Xu, X. (2024). Exploring human-in-the-loop test-time adaptation by synergizing active learning and model selection. Transactions on Machine Learning Research.

Vray, G., Tomar, D., Gao, X., Thiran, J.-P., Shelhamer, E., & Bozorgtabar, B. (2025). ReservoirTTA: Prolonged test-time adaptation for evolving and recurring domains. In Advances in Neural Information Processing Systems (Vol. 38).

Questions for authors
  1. What is the methodological advance beyond HILTTA-style feedback model selection and MORPHEUS-style algorithm selection? Please distinguish candidate objects, information access, and retained learning states, and identify evidence for the claimed advance.

  2. Please provide paired uncertainty estimates across source checkpoints and the class-offset ablation already mentioned in Appendix F. Which conditions favor FORTA over uniform pooling, fixed-share, and convex stacking, after distinguishing class-specific capacity from temporal reweighting?

  3. Do matched-pool results distinguish FORTA from feedback-driven hard selection or input-conditioned gating? A focused comparison, plus a separate weight-dependent-training control, would test the routing and recovery claims. REALDISP is a useful priority given its large main-table gain.

  4. Is a budget- and delay-matched EATTA result available? Otherwise, what incompatibility limits the claim? Please distinguish shared-random-query adaptations from complete systems retaining active queries, and disclose trainable-parameter differences.

Reviewer_8W7POpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces FORTA, an online test-time adaptation framework using sparse, delayed labels. It maintains several source-anchored prediction corrections, trains all supervised corrections on released labels, and dynamically allocates their influence using past predictive performance. The paper presents an information-theoretic lower bound, a constructed allocation advantage, and experiments across image and signal benchmarks.

Critical strengths
  1. The predict-before-query and delayed-feedback protocol is clearly specified and shows great improvement across seven benchmarks.

  2. The matched-portfolio experiments, controlled switching study, negative results, and reproduction discrepancy are reported transparently.

  3. The theoretical results carefully state their assumptions and limitations.

Critical weaknesses
  1. The main paper is difficult to follow without repeatedly consulting the appendix. In particular, it should give a clearer high-level account of the candidate adapters and their objectives, the simulated annotation queue, why image and signal data use different routing objectives, and the motivation for the chosen allocation rules. The current description makes it difficult to identify which parts constitute the core method and which are implementation choices.

  2. The novelty over adaptive online with feedback is proposed by FORTA insufficiently established. Several active test-time adaptation methods (SimATTA [1], EATTA [2], etc.) received feedback to update models, although they also design functions to select samples for labelling compared with FORTA's random sampling. They should be mentioned.

  3. The experiments provide mixed support for FORTA’s allocation rule. Figure 3 shows that uniform pooling is nearly as good or better on shuffled PACS, while fixed-share or convex stacking wins in several PACS and WISDM conditions. Tables 1–2 compare the full portfolio mainly against individual corrections, so they do not isolate whether its gains come from the router, portfolio diversity, or greater aggregate capacity.

[1] Shurui Gui, Xiner Li and Shuiwang Ji, Active Test-Time Adaptation: Theoretical Analyses and An Algorithm, ICLR 2024.

[2] Guowei Wang and Changxing Ding, Effortless Active Labeling for Long-Term Test-Time Adaptation, CVPR 2025

Questions for authors

Please refer to Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.