- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether data weights that help early in training still help at the end. The authors keep the training setup fixed and compare weights selected at early and final checkpoints. On Mind2Web, the early choices can actually make the final loss worse, while terminal selection improves held-out loss. They also check the result with new training seeds, another model, and some Android experiments.
Critical strengths
The main strength is the controlled design. Only the data weights change, and the comparison uses actual training outcomes, so the reversal is unlikely to be an artifact of a bad attribution estimate. The reversal also survives the new-seed check with the same weights, which suggests it is not tied to the seed used for selection.
Critical weaknesses
-
I am not convinced that the cosine results establish local direction rotation. Eq. (5) is about very small perturbations, but every experiment uses h = 0.1. Adding more directions at the same radius doesn’t address this.
-
The sign of the Android “Raw local” baseline is unclear. IIn App. A.2, positive utility means a trajectory helps, but a = −ZᵀDqv seems to point the other way. If Raw local uses +ρu₁, the sign correction may just be undoing a flipped baseline. This only matters for Table 4.
-
The novelty of this paper is low and i did not learn somethong new from this paper.
Questions for authors
-
Could you rerun one existing Mind2Web support with h = 0.05 and check whether the negative cosine and the early-selection reversal still hold?
-
Is Raw local +ρu₁ or −ρu₁, and how does that sign fit the utility definition in App. A.2?
-
Keep consistent wordings throughout the whole paper, and some terms and sentence expressions are used in the paper inconsistently.
If the author could solve these questions, i will improve the score.