- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether local accessibility-tree features improve GUI action selection when candidates and upstream scores are fixed. TopoRerank adds a bounded, instruction-independent residual using a 209-parameter MLP. On AndroidControl, it improves BM25 by 3.62 percentage points and a learned text-only ranker by 0.53 points. The paper also reports area-free matcher controls, structural ablations, an external UGround reference, and positive GUIrilla results after target-domain refitting.
Critical strengths
-
The fixed-candidate protocol isolates final selection from candidate coverage and upstream ranking. Three-seed results, package-level bootstrap analyses, matcher controls, and explicit reporting of null results provide credible support for the narrowly framed empirical claim.
-
The paper includes meaningful alternative explanations and baselines. The area-free BM25 control remains positive, GUIrilla provides a positive refitted result with target node identifiers, and Table A22 compares against bounded linear listwise and pairwise residuals. The authors appropriately qualify transfer and end-to-end applicability.
Critical weaknesses
-
The methodological advance is modest. The paper already discusses structure-aware GUI models and candidate-ranking methods such as RelationAgent, Mind2Web, and RankGround. Its main addition is a bounded MLP correction over local tree and geometry features. The controlled analysis is careful, but the new method and resulting insights offer a limited advance over these established ideas.
-
The benefit becomes less convincing with the stronger upstream ranker: the gain is 0.533 percentage points, with a package-equal confidence interval crossing zero. Table A21 also gives no clear advantage over other structural corrections. Although the head outperforms linear controls under lexical scoring, its usefulness beyond that simpler setting remains insufficiently established.
Questions for authors
-
What is the package-equal gain and 95% confidence interval on top of the learned text-only ranker? A paired comparison with the controls in Table A21 would help interpret the small differences.
-
Table A22 shows an advantage over linear corrections under lexical scoring, but this advantage is less clear with the learned upstream. What do the authors see as the main benefit of the proposed head in that setting?
3 comments
Thank you for the additional experiments. The checkpoint-replay interval for the learned text-only upstream and the new DeBERTa result (+1.31 pp package-equal, [+0.69, +2.03]) show that the correction can still help above a stronger text ranker, and I appreciate the honest reporting. However, the paired comparisons also show no clear advantage of the proposed head over the pointwise-deep control (−0.05 pp, interval includes zero) or over simple linear heads at BM25. This supports my original concern: the main contribution is a bounded structural correction whose benefit can largely be obtained with simpler heads, which is a limited advance over existing work. I will keep my score.
Clarifying the contribution of bounded structural correction
Thank you for considering the additional experiments and recognizing the stronger text-upstream evidence. Your distinction between the structural correction and the particular head architecture helps clarify our contribution. The paired comparisons do not show a clear package-level advantage of the MLP over the pointwise-deep control or the tested linear heads.
We view the contribution as a bounded, instruction-independent structural intervention on frozen candidate scores, together with controlled evidence of when that intervention changes selection. The fitted correction can be attached without retraining the upstream scorer or changing its candidates. Its bound makes the intervention inspectable: an upstream score gap greater than twice the bound cannot be reversed, while close candidates can be reordered using local structure. The simpler bounded heads share this guarantee. At the tested BM25 operating point, the three-feature linear prior recovers most of the observed package-equal gain.
The DeBERTa experiment provides a stronger operating point, with a +1.31 pp package-equal gain [95% CI +0.69, +2.03] over its frozen text scorer. Together with the matched prior and matcher controls, it identifies a setting where structural correction is useful; the choice of head parameterization remains an empirical question. We would welcome your assessment of whether this more precise positioning addresses your concern about the scope of the contribution.
Response to the reviewer: learned upstream and correction controls
Thank you for the careful assessment of the method and alternative explanations. We added a stronger learned upstream and paired comparisons to clarify the contribution.
Requested learned-upstream interval. Replaying the three saved hashed-token checkpoints on GPU reproduces the reported per-seed baseline/corrected hit counts and hit-change counts. The package-equal change is +0.71 pp [95% CI +0.20, +1.25]. The CPU package-equal point estimate agrees within 0.01 pp; row-level runtime differences are described in Appendix H.4. These checkpoint-replay intervals resample 78 packages conditional on fitted checkpoints (revised Table 1).
The submission stated that this interval crosses zero; we cannot verify that statement from the available archived aggregate results. The revised interval is the new checkpoint-replay estimate, not a reconstruction of the original row-level predictions.
Paired comparison on the same learned upstream. We also replayed the saved pointwise-deep control from the submitted Table A21 (revised Table A22) against TopoRerank on the same test rows and frozen hashed-token scores. The CPU checkpoint replay gives MLP minus pointwise-deep −0.05 pp package-equal [95% CI −0.63, +0.50], with a −0.07 pp row-weighted difference over three seeds. The interval includes zero. The pointwise checkpoints reproduce the reported final hit counts; some baseline hit and ranking-change counts differ in the current runtime (revised Appendix H.4).
Benefit above a stronger learned scorer. A MindAct-initialized DeBERTa text scorer fine-tuned on AndroidControl reaches 48.29% top-1; adding the structural correction reaches 50.49%, or +2.20 pp row-weighted and +1.31 pp package-equal [95% CI +0.69, +2.03], across three seeds with fixed candidates and evaluation (revised Table 1). This is a distinct, stronger text-only operating point, not a replacement for the hashed-token setting.
On the hashed-token upstream, the correction has a positive package-equal replay estimate, while the paired comparison does not establish an advantage over the pointwise-deep head. The DeBERTa experiment tests a separate, stronger text-only operating point.
What the correction adds. TopoRerank adds candidate-local structural information with a 209-parameter, instruction-independent head, without retraining the frozen scorer or changing its candidates. Because each correction has magnitude at most B, an upstream score gap greater than 2B cannot be reversed. This explicit bound makes the intervention inspectable: close decisions can change while sufficiently separated pairs retain their order. The same guarantee applies to the bounded pointwise control; the replay does not establish an accuracy advantage for either head. The matched operating-point and matcher controls test when bounded structural correction translates into accuracy gains.
On the hashed-token upstream, the topology-shuffle control gives a +0.03 pp row-weighted gain, compared with +0.53 pp for the unshuffled head. This contrast is consistent with sensitivity to the tested structural features at that operating point.
Relation to the closest work. RelationAgent models spatial, functional, and hierarchical relations in a grounding–relation–navigation pipeline; RankGround ranks screenshot crops for downstream grounding; Mind2Web studies candidate-based web action selection. TopoRerank studies an already-proposed accessibility-tree candidate set: upstream scores and target matching are held fixed while an instruction-independent bounded residual changes the final ordering. This design isolates the contribution of candidate-local structure to selection.
Simple heads and the requested comparisons. At BM25 with the released matcher, the MLP minus a train-fitted linear prior over child count, depth, and area is −0.07 pp package-equal [95% CI −1.19, +0.86]; MLP minus a ten-feature linear head is +0.20 pp [−0.38, +0.74]. These are BM25 comparisons; both intervals include zero. Together with the DeBERTa result, they demonstrate the value of evaluating the structural intervention across operating points and using paired controls to assess head design.
Correction to the tie analysis. We also corrected Appendix F: the learned tie-only gain is approximately +10.04 pp on the 1,195 BM25-tied rows, versus +5.523 pp for smallest-area ordering on those same rows; +2.739 pp used all 4,381 rows as denominator.
The checkpoint interval, paired head comparison, and stronger text-scorer experiment address your questions about the learned upstream, alternative corrections, and the setting in which the head adds value.