- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies supervision placement in finite context adaptation for dynamics prediction. It compares central supervision with midpoint supervision while keeping disagreement regularization fixed. The authors develop a prediction-space criterion to analyze whether supervision-induced changes can be reproduced by hyperparameter retuning, and propose RAA, a retuning-aware method for objective selection under limited computational budgets. Experiments on Lorenz–96 and Burgers systems evaluate the effectiveness of RAA.
Critical strengths
-
The paper highlights an often-overlooked distinction between midpoint supervision during training and the central prediction fitted to observed support at deployment.
-
The two-mode example is convincing. The two modes require different equivalent regularization coefficients, so a single shared coefficient cannot match both simultaneously. This illustrates why the effects of changing supervision cannot always be reproduced by hyperparameter retuning.
-
The experimental design is fairly rigorous. The authors evaluate the methods both with matched hyperparameters and with independent hyperparameter tuning, and also compare their performance under equal computational budgets.
Critical weaknesses
-
RAA estimates prediction responses using a short SGD continuation with a fixed-step gradient adapter, whereas the final training uses LM and Adam. Whether these two procedures produce consistent prediction responses has not been directly validated. An additional consistency analysis would be helpful.
-
The tuning family considered in the diagnostic projection includes only three hyperparameter directions and does not fully align with the search space used in the independent retuning experiments. The authors should clarify the rationale behind this choice and investigate whether the projection residual persists when additional tunable hyperparameters are included.
Questions for authors
-
How closely do the prediction responses estimated by the short SGD continuation agree with those obtained using the final LM/Adam training procedure?
-
What motivates the choice of the three nuisance directions in the diagnostic projection? How sensitive are the residual responses to including additional tuning directions, particularly those used in the independent retuning experiments?