acceptodds
Under review as a conference paper at ICLR 2027

EarlyKnowBench: When Does a Gui Agent Know What the User Wants, and Does It Keep Knowing?

Abstract

A proactive mobile GUI agent must infer user intent from partial interaction trajectories early enough to intervene, while reliably sustaining that prediction over time. However, existing GUI benchmarks evaluate only a single terminal state of a completed trajectory, failing to distinguish a model that discovers and retains the correct goal early from one that reaches it too late to act. To address this limitation, we introduce EarlyKnowBench, a benchmark that probes the latent session goal at every observation step and scores the prediction sequence against an annotated full evidence support time (t*), defined as the earliest step where observed screenshots and executed actions fully specify the goal. Grounding t* in trajectory evidence rather than model-dependent outputs enables standardized, fair comparisons of early intent recognition and temporal retention across sessions and models. We annotate t* across 1,503 sessions comprising 29,630 interaction steps from FingerTip-20K and AndroidControl, evaluating 12 multimodal models across every step. Empirical results reveal that current models struggle to infer user goals prior to t*, and even when successful early on, they systematically fail to retain these predictions across subsequent steps. Furthermore, downstream experiments demonstrate that accurate goal predictions substantially boost action execution success rates. These findings indicate that future proactive agents must look beyond raw scaling to focus explicitly on early intent recognition, temporal prediction retention, and downstream execution alignment.

Primary area:
datasets and benchmarks
Keywords:
early goal prediction, mobile GUI agents, trajectory-level benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S7upOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a prefix-based benchmark for inferring the final goal of a GUI interaction, separating early recognition, consistency over subsequent prefixes, and downstream action quality. It annotates the earliest point at which all goal components are supported by the recorded interaction and evaluates 12 open models on 1,503 FingerTip sessions. The main finding is that an early correct prediction often fails to remain correct as more observations arrive, and improved goal recognition does not reliably translate into better actions.

Critical strengths
  1. The benchmark exposes a useful failure mode that terminal task success obscures. Separating recognition from action quality is consequential: the strongest goal recognizer is not the strongest action predictor. The matched-session analysis around the support boundary in Figure 3 also provides evidence of declining recognition beyond the all-suffix stability metric alone.

  2. The annotation protocol and audits are unusually explicit. The timing and semantic-judgment audits, common evaluation cohort, and None/Gold/Pred goal interventions make the study a useful diagnostic resource. I appreciate the explicit acknowledgement that the support boundary is retrospective and that the action evaluation is offline.

Critical weaknesses
  1. The interpretation of retention is stronger than what the primary metric isolates. Each prefix is evaluated independently without access to earlier predictions (Section 3.1); EarlyStable requires correctness at every remaining prefix. Consequently, Retain@Early mixes prediction consistency with suffix length and a model-dependent subset of sessions that ever yielded an early hit. The cross-scale comparison is therefore not a controlled estimate of improved or impaired memory. Figure 3 partly addresses this concern, but fixed-horizon persistence/transition rates and matched early-hit cohorts are needed to support the scaling interpretation.

  2. The boundary and statistical units limit the early-recognition conclusions. Full support for the final recorded goal is different from that goal being uniquely inferable; a reasonable alternative goal before the boundary may be marked incorrect. The pairwise human audit does not directly validate this ambiguity or the sequence-level stable/unstable labels. In addition, Appendix B.1 acknowledges that the prefix-level action significance tests do not account for within-session dependence. These issues require narrower wording and clustered uncertainty estimates rather than treating every early mismatch or prefix as independent evidence.

Questions for authors
  1. On a common cohort of sessions with an early hit, how do fixed-horizon persistence and correct-to-incorrect transition rates change with model size after stratifying by remaining session length? Please include session- or user-clustered intervals.

  2. Can an audit distinguish final-goal mismatch from genuinely unsupported intent predictions before the boundary, and can the goal-intervention tests be recomputed with session/user clustering? These checks would determine how strongly I interpret the retention and downstream-action claims.

2 comments
AuthorsOpenReview ↗

Comment 1 of 2

Thank you for specifying the controls needed to interpret persistence. We add the common-cohort, remaining-length, and fixed-horizon analyses, together with user-clustered inference.

W1/Q1. What happens after controlling the early-hit cohort and remaining length?

Reviewer question. On a common cohort of sessions with an early hit, how do fixed-horizon persistence and correct-to-incorrect transition rates change with model size after stratifying by remaining session length? Please include session- or user-clustered intervals.

We complement all-suffix stability with three measures: endpoint correctness at t+3 or t+5 conditional on correctness at an early prefix t; the adjacent correct-to-incorrect transition rate; and continuous survival S(h) after the first early hit, with session endings treated as censoring events. Fixed-horizon endpoint correctness allows an intervening error and recovery; continuous survival requires uninterrupted correctness.

Model3-step endpoint correctness5-step endpoint correctnessPost-t* correct → incorrect
Qwen3.5-2B34.527.241.7
Qwen3.5-4B40.933.642.0
Qwen3.5-9B39.231.239.4

We then hold the early-hit cohort fixed: Qwen3.5-2B, 4B, and 9B all recognize the goal early in the same 123 sessions, covering 44 users.

ModelCommon sessionsRetain@EarlyContinuous S(3)Continuous S(5)
Qwen3.5-2B12316.323.79.4
Qwen3.5-4B12314.617.710.5
Qwen3.5-9B12316.315.47.7

We also group these sessions by the number of observations after t* and compare correctness over the same three- or five-step horizon. The table below reports these results together with post-boundary transitions. Appendix D, Table 13 gives 95% percentile intervals from 2,000 user-clustered bootstrap draws.

Observations after t*SessionsBackbone3-step correctness5-step correctnessCorrect → incorrect
0–3542B46.6%29.2%11.8%
0–3544B43.9%35.8%36.8%
0–3549B42.1%42.4%40.9%
4–9392B35.2%28.2%31.3%
4–9394B43.2%49.4%35.7%
4–9399B51.2%36.9%27.8%
≥ 10302B41.8%38.2%26.6%
≥ 10304B54.8%37.0%36.5%
≥ 10309B42.7%38.7%34.3%

The all-suffix Retain@Early counts for this same cohort are:

Observations after t*Common sessions2B Retain@Early4B Retain@Early9B Retain@Early
0–35416/54 (29.6%)15/54 (27.8%)16/54 (29.6%)
4–9393/39 (7.7%)1/39 (2.6%)3/39 (7.7%)
≥ 10301/30 (3.3%)2/30 (6.7%)1/30 (3.3%)

These controls separate early coverage from persistence over comparable horizons. We interpret Retain@Early as prediction persistence under additional observations and report the scale-related coverage gains separately.

The history experiment adds a stateful comparison on all 1,503 sessions for four backbones (Table 19). With history, 4B Retain@Early changes from 14.08% to 21.25%, and Venus from 10.54% to 16.07%; AnyEarly falls from 32.14% to 26.61% and from 39.12% to 27.74%, respectively. EarlyStable increases by 1.14 and 0.33 percentage points on 4B and Venus. The history comparison shows why early coverage and persistence need to be measured separately.

AuthorsOpenReview ↗

Comment 2 of 2

W2/Q2. Can reasonable alternative goals be separated from unsupported predictions, and can action tests account for dependence?

Reviewer question. Can an audit distinguish final-goal mismatch from genuinely unsupported intent predictions before the boundary, and can the goal-intervention tests be recomputed with session or user clustering?

1. Anticipating the final goal from a limited prefix. EarlyKnowBench evaluates whether a model can anticipate the recorded final goal of a session from a limited interaction prefix. The full-support time t* marks the earliest point at which the trajectory supports every required component of that goal. It provides a temporal reference for separating anticipation from recognition with complete supporting evidence. Such anticipation is relevant to proactive assistants that prepare goal-specific help before all goal details become observable.

Two annotators independently reviewed the support boundary in 300 sessions using the original annotation criteria. They checked the evidence for the reference goal and the candidate t*, recording the supporting observations and a written rationale for each decision. Their boundaries agreed exactly in 95.33% of sessions and within one observation in 99.33%. These records provide a human check of the temporal reference used in the evaluation.

Before t*, several goals may be compatible with the available observations. Our early scores measure whether a prediction anticipates the recorded final goal; a mismatch therefore denotes a missed final-goal prediction, without automatically classifying the prediction as unsupported by the current prefix. The 300-session file records boundary evidence and decisions, not a label for every model prediction, so it does not yield a proportion of reasonable alternatives. Correspondingly, Retain@Early measures whether predictions continue to match that same goal as observations accumulate. We interpret early recognition and prediction persistence on this basis.

For example, audit record D155 shows “24,987 ÷ 20” at observation 8; the complete “24,987 ÷ 200” and its preview appear at observation 9. Both humans place t* at 9. The earlier display supports the calculation then visible, while the full recorded calculation becomes supported one observation later. This illustrates why a supported partial or alternative interpretation and a match to the complete recorded goal must be distinguished.

2. Action inference now clusters by user. We compute paired None/Gold/Pred differences within sessions, then jointly flip all sessions of a user in 20,000 randomizations. Holm correction is applied across the 12 models within each contrast. The action differences use 95% percentile intervals from 2,000 user-clustered bootstrap draws; they do not use a normal approximation.

Action contrastModels in this directionSignificant after user clustering + Holm
Gold > None10/126/12
Pred < Gold9/127/12
Pred < None5/121/12

Reference goals improve early action success in 10/12 models, significantly in six after correction. The paired Pred contrasts separately assess the action value of inferred goals. Here, Pred−Gold is the per-session percentage-point difference between the inferred-goal and reference-goal conditions; a negative value means the inferred goal trails the reference goal. These tests account for dependence among sessions from the same user.

The review-analysis appendix adds the fixed-horizon, transition, common-cohort, and length-stratified results with user-clustered intervals. The paper distinguishes reference-goal matching from current-prefix support and uses user-level, multiplicity-corrected action tests.

Reviewer_8VgaOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EarlyKnowBench, a benchmark for inferring users’ goals from partial GUI interaction histories. Using human-recorded trajectories from FingerTip20K and AndroidControl, it annotates when observations fully support the reference goal and evaluates early recognition and prediction stability. Experiments compare general-purpose VLMs and GUI-specialized models, reporting limited early recognition and inconsistent predictions.

Critical strengths
  1. The benchmark evaluates both early goal recognition and prediction stability beyond final-step accuracy.
  2. Human audits support the reliability of the temporal annotations and semantic scoring.
Critical weaknesses
  1. Motivation needs clarification. Please explain when inferring a goal from partial user behavior is more useful than having the user state the goal and letting a GUI agent execute it.

  2. Interpretation of failures needs further clarification. Low early accuracy may reflect limited evidence rather than poor intent understanding. Also, querying each prefix independently does not directly test whether an agent can remember and update a goal. Please clarify these distinctions when discussing early recognition and intent retention.

  3. Task-relevant baselines are missing. Table 2 compares general-purpose VLMs and GUI models, but lacks goal inference baselines. These would help clarify whether the failures come from the models or the evaluation setup.

  4. The reported sample counts and percentages need clarification. Table 2 reports results for 1,503 equally weighted sessions, but the following results for binary session-level metrics are incompatible with integer success counts under this denominator, even after rounding:

    • Qwen3.5-2B: Acc@t (10.22%), AnyEarly (16.11%).
    • UI-Venus-1.5-2B: Acc@t (12.32%) and AnyEarly (18.18%).

    Please clarify the effective denominators, any excluded sessions, and the aggregation procedure.

  5. Table 1 is missing citations for several benchmarks. Please add the corresponding references.

Questions for authors

See Weakness

3 comments
AuthorsOpenReview ↗

Comment 1 of 2

Thank you for the questions about the use case, interpretation, and comparisons. We respond to W1–W5 in order.

W1. When is inferring a goal from user behavior useful?

Reviewer question. When is inferring a goal from partial user behavior more useful than having the user state the goal explicitly and letting the GUI agent execute it?

Our use case is proactive assistance while a user operates their own phone. Recognizing the ongoing goal provides time to prepare relevant information before the user finishes the interaction. EarlyKnowBench measures when this opportunity appears and whether the inferred goal remains correct as observations accumulate.

Two saved predictions illustrate this window; observation indices are zero-based.

Recorded activityCorrect predictionHuman t*Observations remaining after predictionPotential preparation
Checking an undelivered Douyin order9B identifies checking order logistics at observation 5911Prepare a delivery-status summary
Scheduling a Tencent Meeting4B identifies a one-hour meeting at 10:00 the next morning at observation 12165Prepare a calendar comparison before submission

The preparation column describes what the measured window could support. Across models, the median first early hit precedes t* by 2–3 observations. On average, 9.84–12.57 observations remain to the endpoint after this hit. Action tests supply complementary measured evidence. Reference goals improve early action success in 10/12 models.

W2. How do early ambiguity, prediction consistency, and stateful updating differ?

Reviewer question. Low early accuracy may reflect limited evidence rather than poor intent understanding. Also, independent prefix queries do not directly test whether an agent can remember and update a goal. Please separate these interpretations and evaluate stateful updating.

Early matching is evaluated against the recorded goal. t* is the first point supporting every component of that goal. A pre-boundary mismatch can still be a reasonable alternative under the available evidence. We interpret AnyEarly as early reference-goal recognition and Retain@Early as persistence across prefixes.

The additional human review covers 300 sessions. Two annotators independently reviewed candidate t* values using the same support rules as GPT annotation; their exact agreement is 95.33%, and agreement within one observation is 99.33%. Their evidence observations and reasons make the support decisions traceable.

History-conditioned comparisons directly evaluate updating. Sequential History follows each recorded session through AndroidWorld's agent.step interface. Each step receives the current observation prefix, completed actions, and all previous rank-1 goal estimates from that session; history resets between sessions. Independent Prefix receives the same observation prefix without previous predictions. This is stateful trajectory replay with a fixed recorded observation sequence.

The comparison covers Qwen3.5-0.8B, 2B, 4B, and UI-Venus-2-9B. Each setting uses all 1,503 sessions and 29,630 prefixes (Appendix D, Table 19).

BackboneSettingAnyEarlyEarlyStableRetain@EarlyCorrect → incorrect
Qwen3.5-0.8BIndependent Prefix3.330.3310.0029.71 (93/313)
Qwen3.5-0.8BSequential History1.260.4736.8413.64 (27/198)
Qwen3.5-2BIndependent Prefix13.911.4010.0546.49 (451/970)
Qwen3.5-2BSequential History8.252.5931.4521.97 (236/1,074)
Qwen3.5-4BIndependent Prefix32.144.5214.0843.94 (1034/2,353)
Qwen3.5-4BSequential History26.615.6621.2530.82 (733/2,378)
UI-Venus-2-9BIndependent Prefix39.124.1310.5447.86 (1286/2,687)
UI-Venus-2-9BSequential History27.744.4616.0735.28 (912/2,585)

History increases conditional retention and lowers the adjacent correct-to-incorrect rate on all four backbones, while reducing early coverage. EarlyStable increases by 1.14 and 0.33 percentage points on 4B and Venus. History makes early hits less frequent, but predictions that do hit the goal are more likely to stay correct.

AuthorsOpenReview ↗

Comment 2 of 2

W3. What do task-relevant goal-inference baselines show?

Reviewer question. Task-relevant goal-inference baselines are missing from the original comparison. Please add them to clarify whether the failures come from the models or the evaluation setup.

We now compare goal-inference methods on Qwen3.5-4B and UI-Venus-2-9B under fixed source-specific sessions, reference goals, and semantic scoring. Trajectory-CoT reasons over the observed trajectory. AutoToM and LIMP re-rank the same proposal set from Independent Prefix, and JIT induces weighted goals from the observed prefix. These are training-free GUI adaptations. HIM-Agent additionally receives strictly earlier records from the same user, so we identify it as a separate history-enhanced condition.

The table below reports the 4B and Venus comparisons on FingerTip. All prefixes and sessions remain in the score denominators, with invalid outputs scored zero.

BackboneMethodAnyEarlyEarlyStable
Qwen3.5-4BIndependent Prefix32.144.52
Qwen3.5-4BAutoToM19.161.33
Qwen3.5-4BLIMP18.630.67
Qwen3.5-4BJIT6.590.00
Qwen3.5-4BHIM-Agent49.5711.64
Qwen3.5-4BTrajectory-CoT38.323.66
UI-Venus-2-9BIndependent Prefix39.124.13
UI-Venus-2-9BAutoToM24.681.40
UI-Venus-2-9BLIMP24.551.53
UI-Venus-2-9BJIT21.560.93
UI-Venus-2-9BHIM-Agent51.909.45
UI-Venus-2-9BTrajectory-CoT3.730.00

Appendix D, Tables 21–22 report all compared methods on 4B and Venus for FingerTip and AndroidControl. We use the same 32.14% Qwen3.5-4B Independent Prefix baseline throughout the rebuttal and the added method matrices. The 9B comparisons below use the original direct-prediction row as their baseline (Table 20).

SourceMethod (9B)AnyEarlyEarlyStableRetain@EarlyAcc@t*
FingerTipDirect prediction33.604.4613.2721.62
FingerTipTrajectory-CoT36.593.599.8225.15
AndroidControlDirect prediction35.1111.7033.3319.15
AndroidControlTrajectory-CoT38.3010.6427.7824.47

On 4B, Trajectory-CoT raises AnyEarly from 32.14% to 38.32% on FingerTip and from 31.91% to 37.23% on AndroidControl. On Qwen3.5-9B, the completed strategy comparison reaches 36.59% and 38.30%, also improving over direct prediction on both sources. These gains show that the benchmark responds to changes in goal-inference strategy. EarlyStable reports whether the additional early hits persist.

HIM-Agent reaches 49.57% and 51.90% AnyEarly on 4B and Venus, respectively, using additional user history. The Independent/Sequential comparison above isolates the effect of prior predictions while keeping user records fixed.

W4. Why did some binary percentages not match the stated denominator?

Reviewer question. The reported sample counts and percentages were incompatible with integer success counts under the stated 1,503-session denominator. Please clarify the effective denominators, excluded sessions, and aggregation procedure.

We corrected the aggregation to retain all 1,503 frozen sessions, with missing or invalid predictions scored zero and sessions weighted equally. The earlier discrepancy arose because sessions affected by malformed JSON during rollout had been excluded. The table gives the corrected integer counts and percentages.

ModelMetricSuccessful sessionsFixed denominatorCorrected result
Qwen3.5-2BAcc@t*1531,50310.18%
Qwen3.5-2BAnyEarly2421,50316.10%
UI-Venus-1.5-2BAcc@t*1851,50312.31%
UI-Venus-1.5-2BAnyEarly2731,50318.16%

The revised main table and clustered analyses use this corrected rule. Figure 3b is synchronized to the resulting 937-session matched window.

W5. Missing benchmark citations

Reviewer question. Table 1 is missing citations for several benchmark entries. Please add the corresponding references.

Table 1 now cites SNIPS, CLINC150, BANKING77, AndroidControl, FingerTip 20K, AndroidIntent, GUIDE, PIRA-Bench, and Act2Intention.

The revised paper clarifies the proactive use case and support boundary, adds the completed goal-inference and history comparisons, corrects the fixed-denominator counts, and supplies the missing benchmark citations.

AuthorsOpenReview ↗

A Kind Reminder Regarding Our Rebuttal

Dear Reviewer 8Vga,

Thank you again for your valuable comments and feedback on our submission. With one day remaining in the discussion period, we would like to kindly follow up and ask whether our rebuttal has sufficiently addressed your concerns.

Should any questions or concerns remain, we would be grateful for the opportunity to provide further clarification. If you find that our responses have resolved the issues you raised, we would sincerely appreciate your consideration of updating your evaluation scores as appropriate.

Thank you very much for your time and consideration.

Reviewer_RpQkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EarlyKnowBench evaluates proactive mobile GUI agents as a streaming task. Its central device is the full support time t*: the earliest observation at which the recorded screenshots and completed actions support every component of the reference goal (application, operations, entities with roles, constraints), t*_i = min{t : V(r, H_{i,t}) = 1 for every r in R(Y_i)}. Because t* is annotated from the trajectory and hidden from the predictor, it separates extrapolation from partial evidence (correctness before t*) from persistence under later observations (correctness from t* onward). The predictor is queried independently at every prefix and emits five ranked hypotheses that are judged separately, so pre-t* ambiguity is measured rather than penalised; the reference goal is fixed across the session so that Acc_{i,t} scores the same target at every step. Metrics include AUC@t*, Acc@t*, Post-t*, AnyEarly, EarlyStable, Retain@Early = EarlyStable/AnyEarly, and MRR@pre. A paired downstream arm predicts the next action under no goal, the reference goal, or the model's own rank-1 prediction, restricted to t < t*. The benchmark covers 1,503 FingerTip sessions (29,630 prefixes, 66 users) plus 94 AndroidControl sessions, and evaluates 12 models. Main findings: early recognition rarely becomes sustained correctness (AnyEarly exceeds EarlyStable by 7.1x-17.0x across all rows); accuracy often declines after t* (on the 928 sessions with fixed composition, 9 of 12 rows peak exactly at t* and 11 of 12 are lower three observations later); scaling raises AnyEarly 10.7x but Retain@Early only from 10.64% to 13.27%, non-monotonically; and an annotated goal improves pre-t* action success for 10 of 12 rows while an inferred goal underperforms it in 9 and no goal in 5.

Critical strengths
  1. t* is a well-chosen, model-independent reference that separates two behaviours endpoint scoring cannot, and the evidence for this is internally controlled. Because t* is derived from the trajectory rather than from model output, "correct before full support" and "correct from an early step to the end" become separately measurable, and all 12 rows show AnyEarly exceeding EarlyStable by 7.1x to 17.0x. On the 928 sessions supplying a prefix at every offset from t*-3 to t*+3 — a window of fixed composition in which later observations only add evidence — 9 of 12 rows peak exactly at t* and 11 of 12 are lower three steps later (UI-Venus-2-9B 31.36% -> 15.62%; Qwen3.5-9B 22.20% -> 14.66%). Since composition is held fixed, this decline can only be models discarding a correctly formed hypothesis, and the slot decomposition localises the loss to operation (-30.10) and target (-21.19) rather than application (+1.72) — i.e., models retain which app is in use and lose the operation and parameters an assistant would need to act.
  2. The paired downstream design converts an abstract metric into an actionable one. Conditioning the next-action predictor on nothing, the reference goal, or the model's own rank-1 prediction, and restricting to t < t* where no reference action is a completion, isolates the value of a goal without leaking post-t* information or crediting correct terminal stopping. The result that an annotated goal helps 10 of 12 rows while an inferred goal underperforms it in 9 and no goal in 5 — together with the Spearman contrast between Acc@t* and pooled success (0.89) versus pre-t* node accuracy (0.30) — demonstrates concretely that the recognition ranking is not the ranking relevant to acting early. The annotation audits (150 timing sessions, 300 semantic pairs), the single shared judge across all rows, session-equal weighting with 2,000 bootstrap resamples, and the release of a frozen cohort manifest are further positives, as is the explicit statement that the action cohort is selected for readability rather than sampled at random.
Critical weaknesses
  1. The temporal ground truth is model-produced and the human audit cannot establish its correctness, yet every headline finding is a statement about position relative to t*. Table 4 compares the model's t* annotation against a "final human reference" (94.00% exact, 97.33% within one frame), but the paper does not state whether that reference was annotated independently from scratch or obtained by reviewing and correcting model annotations; if the latter, the 94% figure measures agreement with a reference that already contains the model's judgements. Human-human agreement (97.33% exact, MAD 0.047) is computed under the same rubric, and although Appendix A says the stored annotation is withheld during review, the review task still asks annotators to mark each component's first supporting observation under the paper's own definition of support. Since the finding that accuracy declines after full support depends entirely on where t* falls, this is load-bearing: an independent t* annotation on a few hundred sessions, with agreement statistics against the model labels, is needed before the timing claims can be accepted.
  2. Statistical support is thin relative to the claims, and the effective sample size is much smaller than 1,503. The primary cohort comes from a single source (FingerTip) and 66 users, after three filters that retain 1,503 of 5,170 sessions (29%); by the paper's own account the retained sessions are longer than the pool (median 16 observations versus 11, pre-t* interval median 9 versus 6). AndroidControl contributes only 94 sessions and 1,273 prefixes and never appears in a main table. Confidence intervals resample sessions and, as Appendix C concedes, do not account for dependence among sessions of the same user — the dominant correlation structure here. Multiple comparisons are uncorrected (Table 3 reports uncorrected McNemar tests across 12 rows), and the key non-monotone claim about scale rests on differences the paper itself calls "descriptive ... rather than evidence of a statistically established reversal" (Retain@Early 14.08% at 4B versus 13.27% at 9B). A user-clustered bootstrap would likely widen these intervals enough to change what can be concluded.
Questions for authors

Clarifications (answers could change my assessment):

  1. How was the final human t* reference produced — independent annotation from scratch, or review and correction of model annotations? If the latter, please report agreement between two annotators who both annotate without seeing any model output, and the distribution of |t*_human - t*_model|.
  2. All semantic verdicts come from a single Qwen3.8-27B judge, and four of the twelve evaluated models are Qwen3.5. Please re-judge with a non-Qwen judge, report inter-judge agreement on more than 300 pairs, and state whether the ranking of the Qwen3.5 rows changes.
  3. Please report user-clustered confidence intervals for AnyEarly, EarlyStable, and Retain@Early, and a multiple-comparison correction for Table 3. Does "scaling improves coverage but not retention" survive?
  4. AndroidControl is described as a second source but appears only in the appendix. What are AnyEarly, EarlyStable, and Acc@t* on its 94 sessions, and do the Section 5 conclusions hold there?

Suggestions for future work (not affecting my assessment): A blind-prefix study of when a human can infer the hidden intent, to be compared against the record-support annotation (the paper already notes these are separate validation questions); modelling intent that evolves during a session rather than scoring every prefix against a fixed recorded goal; reporting the false-finish rate defined in Section 3.6 but not used in the main tables.

4 comments
AuthorsOpenReview ↗

Comment 1 of 2

Thank you for identifying the support boundary and the paired action evaluation as useful parts of the benchmark. We address the annotation protocol, statistical units, independent judge, and second source below. Percentages in tables are on a 0–100 scale.

W1/Q1. How was the human support boundary produced?

Reviewer question. How was the final human t* reference produced: independent annotation from scratch, or review and correction of model annotations? If the latter, please report agreement between two annotators who both annotate without seeing any model output, and the distribution of |t_human − t*|.

The human reference combines candidate model annotations with human review and correction. For the additional 300-session review (Table 7), two annotators independently checked each candidate t* against the trajectory evidence. The candidate boundary was visible to both annotators. They used the support criteria in the GPT annotation prompt for review and final consensus, with no additional information. The records retain both boundaries, the consensus, evidence observations, and written reasons. This is a targeted boundary-reliability review around the candidate, rather than a blind-from-scratch reannotation; it does not classify every model prediction as a reasonable alternative or as unsupported.

ComparisonExact agreementWithin one observationMean absolute difference
Human 1 vs. Human 295.33%99.33%0.053
Human 1 vs. original t*93.33%96.67%0.150
Human 2 vs. original t*91.33%96.67%0.163
Consensus vs. original t*93.00%97.33%0.140

Here, difference means the absolute gap in observation index, |t_human − t*|. The final human reference is the consensus boundary, so the last row above compares final human t* with the original candidate t*: it matches exactly in 93.00% of sessions and falls within one observation in 97.33%. The full distribution of this gap is below; each row contains all 300 sessions.

ComparisonDifference 0Difference 1Difference 2Difference ≥3
Human 1 vs. Human 22861220
Human 1 vs. original t*2801037
Human 2 vs. original t*2741646
Consensus vs. original t*2791326

The agreement shows consistent application of the support criteria in this review. The support boundary marks when every component of the recorded goal first has observable support. Before that point, the observed prefix may support several possible goals. Our early scores measure whether the model anticipates the recorded final goal.

W2/Q3. Do the findings hold with user clustering and multiple-comparison correction?

Reviewer question. Please report user-clustered confidence intervals for AnyEarly, EarlyStable, and Retain@Early, and a multiple-comparison correction for Table 3. Does “scaling improves coverage but not retention” survive?

We recomputed the FingerTip estimates with 2,000 user-clustered bootstrap draws, resampling users with replacement and retaining all sessions from each sampled user. The reported 95% intervals are percentile intervals from the 2.5th and 97.5th bootstrap percentiles, not normal-approximation intervals. Appendix D, Table 11 reports the uncertainty estimates. The main results are:

ModelAnyEarlyEarlyStableRetain@EarlyAcc@t*
Qwen3.5-0.8B3.130.3310.642.20
Qwen3.5-2B16.102.0012.4010.18
Qwen3.5-4B32.144.5214.0821.76
Qwen3.5-9B33.604.4613.2721.62

From 0.8B to 9B, AnyEarly grows 10.7-fold and EarlyStable 13.4-fold. The user-clustered AnyEarly slope is +9.34 percentage points per parameter doubling, with a 95% percentile interval of +8.26 to +10.46; the Retain@Early slope is +0.84 points, with an interval of −1.99 to +3.32. Thus the coverage trend is clear on this cohort, while retention shows no clear scaling trend. Persistence after an early hit is assessed through the fixed-horizon and common-cohort comparisons reported in the response to S7up.

For the action intervention, we first compute paired differences within sessions. We then jointly flip all sessions from the same user in 20,000 randomizations. Holm correction is applied across the 12 models within each contrast.

Action contrastModels in this directionSignificant after user clustering + Holm
Gold > None10/126/12
Pred < Gold9/127/12
Pred < None5/121/12

Reference goals improve early action success in 10/12 models, with six significant after correction. The Pred comparisons quantify how much of this benefit inferred goals realize. Here, Gold−None, Pred−None, and Pred−Gold are paired percentage-point differences in session-level action success. The δ column in Table 4 is Pred−Gold, so a negative δ means that the inferred goal trails the reference goal.

AuthorsOpenReview ↗

Comment 2 of 2

Q2. Does a non-Qwen judge reproduce the conclusions?

Reviewer question. All semantic verdicts come from a single Qwen3.8-27B judge, and four of the twelve evaluated models are Qwen3.5. Please re-judge with a non-Qwen judge, report inter-judge agreement on more than 300 pairs, and state whether the Qwen3.5 ranking changes.

Gemma-4-31B-it completed re-judging for all 12 models under the submitted credit rule (Appendix D, Table 17). Agreement is computed on common valid rank-1 binary verdicts.

ScopeCommon valid pairsAgreementsAgreement rateCohen's κ
All prefixes355,513332,15093.43%0.598
Early prefixes (t<t*)201,579194,41296.44%0.633

Gemma reproduces the gap between early hits and sustained correctness: AnyEarly exceeds EarlyStable in every model, by 5.1–28.3-fold. Within Qwen3.5, the AnyEarly ordering also remains 9B > 4B > 2B > 0.8B.

The two judges give different orderings on the other metrics. Acc@t* exchanges 4B and 9B. Retain@Early changes from 4B > 9B > 2B > 0.8B to 2B > 0.8B > 4B > 9B. The replicated findings are the early-coverage ordering within Qwen3.5 and the gap between early recognition and sustained correctness.

Q4. What are the complete AndroidControl results?

Reviewer question. AndroidControl is described as a second source but appears only in the appendix. What are AnyEarly, EarlyStable, and Acc@t* on its 94 sessions, and do the Section 5 conclusions hold there?

All 94 sessions, 1,273 prefixes, and 12 models have been evaluated. The complete core table is now in the main paper (Table 3).

ModelAnyEarlyEarlyStableRetain@EarlyAcc@t*
Qwen3.5-0.8B6.380.000.006.38
Qwen3.5-2B21.283.1915.0012.77
Qwen3.5-4B29.799.5732.1427.66
Qwen3.5-9B35.1111.7033.3319.15
UI-Voyager2.130.000.000.00
GUI-Owl-1.5-2B13.836.3846.1511.70
MAI-UI-2B14.894.2628.5712.77
UI-Venus-1.5-2B17.024.2625.0017.02
UI-TARS-1.5-7B14.895.3235.7115.96
UI-Venus-1.5-8B30.8511.7037.9324.47
UI-Mate-9B41.4913.8333.3334.04
UI-Venus-2-9B35.1111.7033.3327.66

Qwen3.5 AnyEarly increases from 6.38% to 35.11%; the 9B Retain@Early is 33.33%. The AndroidControl intervals in Appendix D are 95% percentile intervals from 2,000 session-clustered bootstrap draws. Every model has more early hits than sustained early successes on this source as well.

We also account for the different suffix lengths: t* is the final observation in 54/94 AndroidControl sessions, compared with 240/1,503 FingerTip sessions. Reweighting FingerTip to AndroidControl's post-boundary length distribution changes 9B Retain@Early from 13.3% to 25.6%. The revised appendix reports this length control alongside the complete table.

The revised paper adds the second-source core results. The review-analysis appendix includes the additional human boundary review, Gemma re-judging, user-clustered recognition estimates, and corrected action tests.

Thanks for the detailed rebuttal that addressed most of my concerns. I would like to keep my rating as weak accept.

Thank you very much for your positive assessment of our work! We are delighted that our rebuttal has addressed your concerns. We sincerely appreciate your time and thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for the constructive feedback. The revision focuses on annotation reliability, prediction persistence, and task-relevant comparisons.

ReviewerMain concerns (very brief)What we did in rebuttal
RpQkReliability of the support time, user-level uncertainty, multiple comparisons, judge dependence, and the second sourceAdded the 300-session boundary review, user-clustered percentile intervals, user-level permutation tests with Holm correction, an independent judge, and complete AndroidControl results.
8VgaUsefulness of early goal inference, ambiguity before full support, stateful updating, baselines, denominators, and citationsAdded concrete proactive-assistance examples, the four-backbone history comparison, 4B and Venus method matrices, fixed-denominator counts, and the missing references.
S7upPersistence after an early hit, cohort and length controls, alternative goals, and dependence in the action testAdded fixed-horizon, common-cohort, length-stratified, and history analyses; clarified final-goal matching; and reran the action test with user clustering and Holm correction.

The recognition intervals use 2,000 cluster bootstrap draws. Users are resampled with replacement while all sessions from a sampled user stay together; AndroidControl uses session clusters because it has no user map. The reported 95% intervals are percentile intervals between the 2.5th and 97.5th bootstrap percentiles, without a normal approximation.

The action test uses 20,000 user-level sign flips on per-session differences, with all sessions from one user flipped together. Holm correction is applied across the 12 models within each contrast.

We understand that this year’s ICLR review process is especially demanding, and we are sincerely grateful that you took the time to read our detailed rebuttal and additional experiments. We remain confident that EarlyKnowBench offers a meaningful contribution to the study of proactive mobile GUI agent!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.