- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the parameter states used by individual updates affect finite-horizon adaptation. The authors introduce parameter-state evaluation graphs to represent these dependencies.
The paper shows that different update schedules can produce different results even when they share the same updates, order, and aggregate delay statistics. An exact affine analysis explains this effect.
The authors also propose a Fresh-reference predictor. It estimates the endpoints of alternative evaluation graphs using a single sequential reference trajectory. Experiments across several adaptation settings demonstrate the importance of read-state assignment and evaluate the predictor's accuracy and practical utility.
Critical strengths
- A well-defined research question supported by theory and controlled experiments.
The paper identifies an important limitation of aggregate staleness measures. The matched-delay comparisons isolate the effect of assigning stale reads to different updates.
The exact affine analysis provides a clear explanation for these differences. The empirical results further support the theoretical findings. This makes the central contribution convincing.
- Careful evaluation of the proposed predictor and its limitations.
The Fresh-reference predictor improves ranking performance in the primary supervised setting. For example, the median Spearman correlation increases from 0.569–0.616 with local defects to 0.722–0.772 with pathwise transport.
The authors also examine different datasets, model parameters, objectives, and adaptation horizons. Importantly, they report cases where prediction or graph selection becomes less effective. These experiments strengthen the credibility of the findings.
Critical weaknesses
- The practical impact of evaluation graph selection is sensitive to calibration.
The primary experiments show substantial differences between evaluation graphs. However, these differences become much smaller after controlling for initialization and graph-specific learning rates.
Table 4 shows that the Fresh–common-origin contrast decreases to 0.002–0.010 CE under joint calibration. Output calibration also removes more than 92% of the raw CE contrast.
The structural differences remain, but their practical importance under well-calibrated adaptation is less clear.
- The predictor does not yet provide a clear computational advantage.
The predictor improves endpoint ranking without executing every candidate graph. However, the current implementation is slower than exhaustive selection in the reported K=5 experiment.
Furthermore, the predictor-guided selector does not meet the stated noninferiority criterion relative to exhaustive selection. Its advantages also become very small after output calibration.
These results limit the practical benefit of the proposed prediction method.
- The predictor's effectiveness varies substantially across adaptation settings.
The predictor works well in the primary supervised setting, but its performance does not transfer consistently. For example, its pooled median rank correlation falls to 0.067 in the CLIP-LoRA setting.
The main experiments also focus on short horizons with contiguous refresh schedules and memoryless updates. It remains unclear how well the predictor extends to more general update dependencies and optimizers with internal states.
Questions for authors
- Under what conditions does evaluation graph selection provide meaningful benefits after both learning-rate and output calibration?
- Could the computational cost of the Fresh-reference predictor be reduced enough to outperform exhaustive graph evaluation?
- What explains the large reduction in predictor accuracy in the CLIP-LoRA setting? Can the proposed reliability diagnostics help identify when prediction should not be used?
- How difficult would it be to extend the evaluation graph framework to non-contiguous schedules or optimizers such as Adam?
3 comments
Reply to reviewer omee (part 2 of 2)
"What explains the large reduction in predictor accuracy in the CLIP-LoRA setting? Can the proposed reliability diagnostics help identify when prediction should not be used?"
Prediction fidelity tracks the size of the parameter displacements, and step-size calibration restores accurate ranking in CLIP-LoRA. The predictor propagates stale-read defects along the sequential reference trajectory. At the calibrated step size the support-side diagnostics in CLIP-LoRA are 39× (κ₀) and 103× (D_G) larger than in the supervised setting, consistent with stale reads moving the parameters far from that trajectory. Matching the step size to this regime is what works; block-recursive transport and a quadratic correction did not help. On a new bank of 80 episodes across four datasets, reducing the step size from 0.03 to 0.0075 raises the mean Spearman correlation from 0.090 to 0.506 [0.417, 0.590], full transport beats local defects by +0.206 [0.118, 0.291], and the adapted model improves over no adaptation by 0.725 CE (positive in 77 of 80 episodes). D_G tracks the recovery across episodes (Spearman −0.71) and is largest on EuroSAT, which has the lowest ranking (0.26, against 0.58–0.61 on the other three; Appendix B.5).
The diagnostics also identify this regime from support data alone. The pre-specified κ₀ and D_G track per-episode error (worst-decile AUC 0.91–0.94), and at the calibrated step size all 800 CLIP-LoRA episodes have κ₀ above the largest supervised value; a threshold set on supervised episodes would therefore abstain on every one of them. Within the supervised setting, a κ₀ threshold fixed on a development bank flags 17–20% of new episodes, whose prediction error is 3.2–3.5× that of the remaining ones (post hoc; Appendix A.22).
"How difficult would it be to extend the evaluation graph framework to non-contiguous schedules or optimizers such as Adam?"
The representation already covers both cases, and the predictor needs one additional specification for each. The evaluation graph already allows any causal read assignment (update t reads some earlier state a(t) < t), and the predictor's ingredients, namely the stale defect of update t at state a and its transport along the reference path, are defined for every pair a < t; contiguous schedules are the special case in which a(t) is a block start. For non-contiguous schedules, one additionally needs to fix the commit rule, that is, which updates are applied together and when. For Adam, the state must include the moment estimates, and one must specify whether an update reads stale moments in addition to stale parameters; the defect and transport then act on this extended state. Regarding horizon, the revision already goes beyond five updates: on the complete 128-graph family at K=8, the graded effect persists (Holm p = 0.0012), the predictor improves rank correlation over local defects, and support-only selection gains +0.157 to +0.219 CE; on the complete 2,048-graph family at K=12, the cached predictor ranks graphs with median Spearman 0.55 (local defects 0.31) while running 18–19× faster than execution. Our experiments use contiguous schedules and memoryless updates, and we see the two extensions above as the natural next steps.
We are grateful for your support of this work. We hope the revision further strengthens your confidence in it, and we remain happy to discuss any further questions.
Additional results on general causal schedules and adaptive prediction
Thank you for your constructive questions. We have extended the revised manuscript with new results addressing two points from your review.
Extension to non-contiguous schedules. We show that the Fresh-reference predictor applies to general causal read assignments using the same defect and transport formulation. We evaluate 24 non-contiguous schedules across 50 confirmation episodes under ResNet-12 feature-based adaptation, updating a 2,565-parameter linear head on frozen backbone features.
At the calibrated step size, the predictor achieves median Spearman 0.977 and within-budget R² 0.960, compared with 0.807 and 0.465 for the initialization-anchored second-order baseline. The same cached computation supports these schedules without additional model evaluations for each candidate graph (Appendix B.10, Table 46). Extending the method to optimizers with internal state, such as Adam, remains a separate direction.
Adaptive prediction and selection. We also turn the support-side reliability diagnostics into an adaptive predictor. Across 1,200 confirmation cases, its mean ranking correlation reaches 0.7985, above both fixed predictors. Under the two-block selection budget (B=2), the adaptive predictor reduces held-out query CE by 0.0276 relative to Always-Full on 300 strong-42 episodes. A lightweight variant separately reduces prediction-stage GPU latency by 35.9% while improving ranking over both baselines (Appendix A.22, Table 32).
These additions complement the previously reported mechanism analysis, K=12 caching results, and CLIP-LoRA step-size experiments. We appreciate the questions that motivated these extensions.
Reply to reviewer omee (part 1 of 2)
Thank you for the encouraging assessment, and for noting that the central contribution is convincing and that reporting the cases where prediction or selection becomes less effective strengthens “the credibility of the findings”. Your four questions shaped several of the additions in this revision, and we address them in order below.
"Under what conditions does evaluation graph selection provide meaningful benefits after both learning-rate and output calibration?"
Selection matters most when the step size is fixed and the refresh budget is limited. The answer depends on whether the step size can be re-tuned for each graph. If it can, re-tuning closes almost all of the fresh–batched gap, and selection retains only a small gain. If the step size is fixed, for instance when a meta-learned step size is deployed without per-graph re-tuning, and the number of refreshes is budgeted, selection matters considerably more. In both cases, PJGS uses support data only (Appendix A.12, Table 16):
Tuning also leaves in place the structure on which selection acts: matched-delay contrasts still vary by 0.034–0.086 CE across support orders within an episode, and the predictor ranks them with Spearman 0.938–0.996 (Table 3). In short, tuning reduces how much there is to gain from choosing a graph, but not the differences between graphs themselves.
A third condition is that graphs are scored under the same initialization convention they will run from. PJGS keeps the two consistent by design: the fold heads and the executed head both start from S1 prototypes. A new controlled experiment shows why this matters (Appendix A.12). When execution starts from the full support set but scoring keeps S1, selection falls below the best fixed graph; scoring each fold from all support images outside its validation set restores it, lowering held-out CE by 0.127 and 0.112 on two checkpoints (100 new episodes each) and raising top-1 graph-selection accuracy among four graphs from 32% to 50% and from 30% to 44%. Parameter-state conventions therefore shape the choice of a graph, not only its endpoint.
"Could the computational cost of the Fresh-reference predictor be reduced enough to outperform exhaustive graph evaluation?"
Yes: computing the shared quantities once makes prediction faster than exhaustive evaluation for complete graph families. The submitted code recomputed every stale defect and its transport for each candidate graph, although both depend only on the reference trajectory and on the (receiving update, read state) pair. The cost therefore came from this recomputation rather than from the method itself. Computing these quantities once reproduces the submitted predictions bit for bit and reverses the comparison (Table 40):
These cached predictions also remain accurate at this scale: on all 2,048 graphs at K=12, each executed for ground truth on 50 new episodes, they rank the graphs with median Spearman 0.55, against 0.31 for local defects (Table 41).
Only in the ten-candidate K=5 selection protocol, where PG-JGS already shared these quantities, is directly running such a small family still faster (1.52 s vs. 1.92 s median). Beyond such small families, execution grows as K·2^(K−1) operator applications, while the cache requires O(K²) field evaluations and O(K³) JVPs, plus inexpensive vector additions, so the advantage of prediction widens with K (operation counts are given in our reply to aA49).