- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a prefix-based benchmark for inferring the final goal of a GUI interaction, separating early recognition, consistency over subsequent prefixes, and downstream action quality. It annotates the earliest point at which all goal components are supported by the recorded interaction and evaluates 12 open models on 1,503 FingerTip sessions. The main finding is that an early correct prediction often fails to remain correct as more observations arrive, and improved goal recognition does not reliably translate into better actions.
Critical strengths
-
The benchmark exposes a useful failure mode that terminal task success obscures. Separating recognition from action quality is consequential: the strongest goal recognizer is not the strongest action predictor. The matched-session analysis around the support boundary in Figure 3 also provides evidence of declining recognition beyond the all-suffix stability metric alone.
-
The annotation protocol and audits are unusually explicit. The timing and semantic-judgment audits, common evaluation cohort, and None/Gold/Pred goal interventions make the study a useful diagnostic resource. I appreciate the explicit acknowledgement that the support boundary is retrospective and that the action evaluation is offline.
Critical weaknesses
-
The interpretation of retention is stronger than what the primary metric isolates. Each prefix is evaluated independently without access to earlier predictions (Section 3.1); EarlyStable requires correctness at every remaining prefix. Consequently, Retain@Early mixes prediction consistency with suffix length and a model-dependent subset of sessions that ever yielded an early hit. The cross-scale comparison is therefore not a controlled estimate of improved or impaired memory. Figure 3 partly addresses this concern, but fixed-horizon persistence/transition rates and matched early-hit cohorts are needed to support the scaling interpretation.
-
The boundary and statistical units limit the early-recognition conclusions. Full support for the final recorded goal is different from that goal being uniquely inferable; a reasonable alternative goal before the boundary may be marked incorrect. The pairwise human audit does not directly validate this ambiguity or the sequence-level stable/unstable labels. In addition, Appendix B.1 acknowledges that the prefix-level action significance tests do not account for within-session dependence. These issues require narrower wording and clustered uncertainty estimates rather than treating every early mismatch or prefix as independent evidence.
Questions for authors
-
On a common cohort of sessions with an early hit, how do fixed-horizon persistence and correct-to-incorrect transition rates change with model size after stratifying by remaining session length? Please include session- or user-clustered intervals.
-
Can an audit distinguish final-goal mismatch from genuinely unsupported intent predictions before the boundary, and can the goal-intervention tests be recomputed with session/user clustering? These checks would determine how strongly I interpret the retention and downstream-action claims.
2 comments
Comment 2 of 2
W2/Q2. Can reasonable alternative goals be separated from unsupported predictions, and can action tests account for dependence?
Reviewer question. Can an audit distinguish final-goal mismatch from genuinely unsupported intent predictions before the boundary, and can the goal-intervention tests be recomputed with session or user clustering?
1. Anticipating the final goal from a limited prefix. EarlyKnowBench evaluates whether a model can anticipate the recorded final goal of a session from a limited interaction prefix. The full-support time t* marks the earliest point at which the trajectory supports every required component of that goal. It provides a temporal reference for separating anticipation from recognition with complete supporting evidence. Such anticipation is relevant to proactive assistants that prepare goal-specific help before all goal details become observable.
Two annotators independently reviewed the support boundary in 300 sessions using the original annotation criteria. They checked the evidence for the reference goal and the candidate t*, recording the supporting observations and a written rationale for each decision. Their boundaries agreed exactly in 95.33% of sessions and within one observation in 99.33%. These records provide a human check of the temporal reference used in the evaluation.
Before t*, several goals may be compatible with the available observations. Our early scores measure whether a prediction anticipates the recorded final goal; a mismatch therefore denotes a missed final-goal prediction, without automatically classifying the prediction as unsupported by the current prefix. The 300-session file records boundary evidence and decisions, not a label for every model prediction, so it does not yield a proportion of reasonable alternatives. Correspondingly, Retain@Early measures whether predictions continue to match that same goal as observations accumulate. We interpret early recognition and prediction persistence on this basis.
For example, audit record D155 shows “24,987 ÷ 20” at observation 8; the complete “24,987 ÷ 200” and its preview appear at observation 9. Both humans place t* at 9. The earlier display supports the calculation then visible, while the full recorded calculation becomes supported one observation later. This illustrates why a supported partial or alternative interpretation and a match to the complete recorded goal must be distinguished.
2. Action inference now clusters by user. We compute paired None/Gold/Pred differences within sessions, then jointly flip all sessions of a user in 20,000 randomizations. Holm correction is applied across the 12 models within each contrast. The action differences use 95% percentile intervals from 2,000 user-clustered bootstrap draws; they do not use a normal approximation.
| Action contrast | Models in this direction | Significant after user clustering + Holm |
|---|---|---|
| Gold > None | 10/12 | 6/12 |
| Pred < Gold | 9/12 | 7/12 |
| Pred < None | 5/12 | 1/12 |
Reference goals improve early action success in 10/12 models, significantly in six after correction. The paired Pred contrasts separately assess the action value of inferred goals. Here, Pred−Gold is the per-session percentage-point difference between the inferred-goal and reference-goal conditions; a negative value means the inferred goal trails the reference goal. These tests account for dependence among sessions from the same user.
The review-analysis appendix adds the fixed-horizon, transition, common-cohort, and length-stratified results with user-clustered intervals. The paper distinguishes reference-goal matching from current-prefix support and uses user-level, multiplicity-corrected action tests.
Comment 1 of 2
Thank you for specifying the controls needed to interpret persistence. We add the common-cohort, remaining-length, and fixed-horizon analyses, together with user-clustered inference.
W1/Q1. What happens after controlling the early-hit cohort and remaining length?
Reviewer question. On a common cohort of sessions with an early hit, how do fixed-horizon persistence and correct-to-incorrect transition rates change with model size after stratifying by remaining session length? Please include session- or user-clustered intervals.
We complement all-suffix stability with three measures: endpoint correctness at t+3 or t+5 conditional on correctness at an early prefix t; the adjacent correct-to-incorrect transition rate; and continuous survival S(h) after the first early hit, with session endings treated as censoring events. Fixed-horizon endpoint correctness allows an intervening error and recovery; continuous survival requires uninterrupted correctness.
We then hold the early-hit cohort fixed: Qwen3.5-2B, 4B, and 9B all recognize the goal early in the same 123 sessions, covering 44 users.
We also group these sessions by the number of observations after t* and compare correctness over the same three- or five-step horizon. The table below reports these results together with post-boundary transitions. Appendix D, Table 13 gives 95% percentile intervals from 2,000 user-clustered bootstrap draws.
The all-suffix Retain@Early counts for this same cohort are:
These controls separate early coverage from persistence over comparable horizons. We interpret Retain@Early as prediction persistence under additional observations and report the scale-related coverage gains separately.
The history experiment adds a stateful comparison on all 1,503 sessions for four backbones (Table 19). With history, 4B Retain@Early changes from 14.08% to 21.25%, and Venus from 10.54% to 16.07%; AnyEarly falls from 32.14% to 26.61% and from 39.12% to 27.74%, respectively. EarlyStable increases by 1.14 and 0.33 percentage points on 4B and Venus. The history comparison shows why early coverage and persistence need to be measured separately.