acceptodds
Under review as a conference paper at ICLR 2027

SoGUI: Cognition-Conditioned Interaction for Social GUI Agents

Abstract

Progress in vision–language models has made GUI agents increasingly capable, yet most benchmarks provide the interaction goal and primarily evaluate where to act. Social interfaces introduce an earlier challenge: the agent must infer an interaction-relevant semantic state from observed content before a specified policy determines whether and how to interact. We introduce SoGUI, a cognition-conditioned interaction (CCI) formulation that enables separate analysis of semantic inference and policy-conditioned GUI grounding under a controlled policy. We further construct a 3,916-record Theory-Grounded Social-Semantic Cognition Benchmark covering affective valence, visible source-credibility cues, and gain/loss framing, with image-disjoint held-out-platform evaluation. Across three training seeds, SoGUI-SFT-CoG improves Strict Accuracy over the standard cognition-free NoCoG baseline by 6.56/12.37 points on ID/OOD and achieves 94.02%/87.21% E2E-Hit. A prompt-matched NoCoG+Rule control shows that explicit cognition supervision provides additional gains beyond policy-rule prompting alone, improving Strict Accuracy by 2.69/4.73 points on ID/OOD. Error decomposition further shows that once semantic inference is improved, GUI localization becomes the dominant residual bottleneck, particularly under held-out-platform evaluation. These results indicate that semantic cognition and GUI grounding are partially separable capabilities, motivating GUI agents that combine stronger semantic understanding with robust cross-platform grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, multimodal agents, GUI grounding, cognition-conditioned interaction, vision-language models, social media, reinforcement learning, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_8dxqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates semantic-state inference and GUI grounding in social GUI agents, arguing that these are partially separable capabilities. It introduces a benchmark with 3,916 records from six platforms, covering affective valence, source-credibility cues, and message framing. For each task, the agent must infer the semantic state, determine the action under a predefined policy, and locate the corresponding GUI element when an interaction is required. The paper evaluates GUI-specialized models, general-purpose VLMs, and its SoGUI models. The reference SoGUI model achieves 94.02% E2E-Hit on ID evaluation and 87.21% on held-out platforms. Explicit semantic-state supervision improves Strict Accuracy over the prompt-matched baseline by 2.69 and 4.73 percentage points, respectively. Finally, the authors apply GRPO to address the remaining localization errors.

Critical strengths

1- The paper studies a well-motivated problem. GUI agents need to understand the content before deciding whether and how to interact. The benchmark provides separate annotations for semantic states, policy-derived actions, and GUI locations, making it possible to examine where interaction failures occur. This is a useful contribution toward understanding capabilities that conventional grounding accuracy alone does not capture.

2- The controlled supervision comparison provides useful evidence. The CoG and NoCoG+Rule conditions use the same training data and policy instructions, allowing the authors to examine the benefit of explicitly supervising the semantic state. The reported Strict Accuracy improvements are consistent across three training seeds, averaging 2.69 percentage points on ID and 4.73 points on held-out platforms. This strengthens the evidence that explicit semantic supervision can improve cognition-conditioned interaction.

Critical weaknesses

1- The benchmark substantially simplifies the semantic understanding required in real social interactions. Real social content can involve mixed emotions, uncertain credibility, and context-dependent meanings, with several factors jointly influencing an interaction decision. The benchmark represents these judgments through seven predefined labels across three separate tasks, each mapped to a fixed action. This makes it unclear how well the reported improvements extend to situations where semantic states are ambiguous or cannot be captured by these categories. Consequently, the broader significance of the contribution for realistic social GUI interaction remains insufficiently established.

2- The model comparisons may confound semantic understanding with instruction following. In Figure 6, POINTS-GUI-G outputs the alternatives “gain or loss” and “like or no-action” instead of selecting valid labels. This raises the possibility that some failures of specialized GUI agents reflect instruction-following or output-format difficulties. Without quantifying these failures separately, it is difficult to attribute their lower scores specifically to weaker semantic understanding. This makes the interpretation of the cross-model capability comparisons less conclusive.

Questions for authors

1- Dataset clarification: How were posts with mixed emotions, uncertain credibility, or ambiguous framing handled during dataset construction? Were these cases excluded or assigned a single label, and approximately what proportion of the collected content did they represent? This would help clarify how much semantic complexity the benchmark captures.

2- Evaluation clarification: For the GUI-specialized agents, how frequently do failures involve invalid or copied output labels, as illustrated in Figure 6? Could you separate these failures from valid but incorrect semantic predictions, and report semantic accuracy on valid outputs alongside the original scores? This would help assess how much instruction-following difficulties affect the capability comparison.

3- Results clarification: Could you report the CoG versus NoCoG+Rule improvements in E2E-Hit and Strict Accuracy separately for each cognition dimension and held-out platform? This would clarify whether the aggregate gains reflect consistent improvements or are concentrated in particular tasks or interfaces.

5 comments

Response to Reviewer 8dxq W1:

We thank the reviewer for this important concern. We agree that real-world social interactions involve semantic ambiguity, mixed emotions, and multiple interacting factors. We would like to clarify the distinction between the general CCI capability and its controlled benchmark instantiation.

1. Rationale for the controlled design. SoGUI evaluates whether GUI agents can infer interaction-relevant semantic states from observed content and execute a specified interaction policy. Affective valence, visible source credibility, and message framing are theoretically grounded, interaction-relevant semantic constructs, rather than an exhaustive taxonomy of social semantics. These dimensions and deterministic mappings instantiate CCI under controlled conditions; they do not define the limits of the general formulation.

This design enables separate diagnosis of semantic inference, policy-consistent action selection, and GUI grounding. Thus, the benchmark serves as a diagnostic instrument for content-conditioned interaction, rather than a comprehensive simulation of real-world social decision-making.

2. Generalization beyond fixed policies. To examine whether models merely memorize the original semantic-to-action mappings, we evaluated the seed-42 CoG and NoCoG+Rule models under changed test-time policies without further fine-tuning. We reversed the emotion and framing mappings and cyclically permuted the credibility mapping, keeping screenshots and ground-truth semantic labels unchanged.

ModelID Action Acc.OOD Action Acc.
NoCoG+Rule80.14%83.51%
CoG93.67%89.83%
Improvement+13.53 pp+6.32 pp

The screenshot-clustered paired-bootstrap 95% confidence intervals are [9.87, 17.43] pp (ID) and [3.30, 9.23] pp (OOD). These results provide evidence that the benefit of explicit semantic supervision extends beyond the original fixed action mappings.

3. Scope and limitations. We acknowledge that the current benchmark does not cover ambiguous semantic states or multi-factor social decisions. Our claim is not that SoGUI captures the full complexity of social interaction, but that it establishes a reproducible diagnostic evaluation of a fundamental capability for content-conditioned GUI interaction: inferring task-relevant semantic states from observed content to guide interaction. We will clarify the distinction between the general CCI formulation and its controlled benchmark instantiation, and explicitly discuss semantic ambiguity and multi-factor decisions as limitations and future extensions.

Response to Reviewer 8dxq W1:

We thank the reviewer for identifying this potential confound. We agree that instruction-following and output-format failures should not be conflated with semantic understanding errors. We conducted additional analyses to separate these failure sources.

1. Separating invalid outputs from semantic errors. We audited the original predictions and distinguished invalid cognition outputs (e.g., copied alternatives or missing labels) from valid but incorrect semantic predictions.

Table 1. Original prediction error decomposition.

ModelSplitInvalid CognitionValid but IncorrectAccuracy on Valid Outputs
UI-R1-EID37.08%5484.92%
UI-R1-EOOD38.67%7082.41%
GUI-R1-7BID7.21%5988.83%
GUI-R1-7BOOD4.16%11082.32%
POINTS-GUI-GID17.75%10278.21%
POINTS-GUI-GOOD19.88%12076.92%
ScaleCUA-7BID21.44%4390.38%
ScaleCUA-7BOOD19.26%7585.69%

For POINTS-GUI-G, all invalid cognition outputs contain unselected alternatives, as illustrated in Figure 6. However, semantic errors remain among valid outputs, confirming that both failure types contribute to performance.

2. Prompt-and-protocol adaptation. We additionally evaluated UI-R1-E using clarified task instructions and a native action-coordinate interface, without task-specific fine-tuning. This combined adaptation tests protocol sensitivity without isolating individual modifications.

Table 2. UI-R1-E before and after protocol adaptation.

MetricOriginal IDAdapted IDOriginal OODAdapted OOD
Valid cognition outputs62.92%99.65%61.33%99.38%
Overall semantic accuracy53.43%84.71%50.54%83.20%
Accuracy on valid outputs84.92%85.01%82.41%83.72%

The adaptation substantially increases output validity and overall semantic accuracy, while accuracy among valid outputs changes only marginally. This suggests that the overall improvement is largely associated with increased output validity, although valid-but-incorrect semantic predictions remain.

Together, these results reveal distinct failure sources: output-protocol violations and incorrect semantic predictions. We agree that the specialists' lower end-to-end performance should not be attributed exclusively to semantic understanding. We will revise Findings 1 and 2 accordingly, distinguishing original-protocol performance from semantic accuracy on valid outputs and explicitly acknowledging instruction-following and protocol sensitivity.

Response to Reviewer 8dxq Q1:

Thank you for this question. Posts that did not admit a single defensible label under the task-specific annotation criteria were excluded rather than force-labeled. We retained cases with sufficiently observable evidence in the screenshot and accompanying text, without relying on external context. Clearly mixed affective signals, insufficient source-credibility cues, and indeterminate gain/loss framing did not meet the inclusion criteria.

For retained samples, annotators assigned task-specific semantic labels using predefined criteria: affective language and visual cues for valence, visible account identity and verification cues for credibility, and gain/loss emphasis for framing. Annotation disagreements were resolved through adjudication before finalizing the labels.

We conducted an independent annotation-quality audit with three annotators on 500 class-balanced samples per dimension:

DimensionSamplesUnanimous AgreementFleiss' κ
Affective valence50091.4%0.886
Source credibility50084.2%0.833
Message framing50081.4%0.764

These results assess label reliability within the retained benchmark, not the prevalence of semantic ambiguity in the original collection. The available construction records do not preserve a verified ambiguity-specific exclusion count, so we cannot reliably estimate the proportion of excluded ambiguous posts.

Accordingly, the benchmark focuses on controlled, single-label cases supported by observable task-relevant evidence, rather than representing the full distribution of social-media content or evaluating ambiguous semantic judgments. This controlled scope enables reproducible evaluation and clearer attribution of semantic inference and GUI grounding errors. We will clarify the inclusion criteria, adjudication procedure, and scope limitations in the revised paper.

Response to Reviewer 8dxq Q2:

We thank the reviewer for this suggestion. We audited the original main-table predictions using the original semantic-field parsing rules, distinguishing invalid cognition outputs (e.g., copied alternatives, missing labels, or labels outside the allowed set) from valid but incorrect semantic predictions. ID and OOD contain 569 and 649 records, respectively.

ModelSplitInvalid Cognition, n (%)Valid but Incorrect, nOverall Semantic Acc.Valid-Output Acc.
UI-R1-EID211 (37.08%)5453.43%84.92%
UI-R1-EOOD251 (38.67%)7050.54%82.41%
GUI-R1-7BID41 (7.21%)5982.43%88.83%
GUI-R1-7BOOD27 (4.16%)11078.89%82.32%
POINTS-GUI-GID101 (17.75%)10264.32%78.21%
POINTS-GUI-GOOD129 (19.88%)12061.63%76.92%
ScaleCUA-7BID122 (21.44%)4371.00%90.38%
ScaleCUA-7BOOD125 (19.26%)7569.18%85.69%

Overall semantic accuracy uses all records as the denominator, whereas valid-output accuracy is conditional on a single allowed cognition label being extracted.

For POINTS-GUI-G, all 101 ID and 129 OOD invalid cognition outputs contain unselected alternatives, consistent with Figure 6. Its invalid action outputs occur in 85/569 ID cases (14.94%) and 71/649 OOD cases (10.94%). Thus, output-protocol failures and valid-but-incorrect semantic predictions are both substantial.

We additionally evaluated UI-R1-E with clarified instructions and a native action-coordinate interface, without task-specific fine-tuning:

SplitSettingValid CognitionInvalid, nValid but Incorrect, nOverall Semantic Acc.Valid-Output Acc.
IDOriginal62.92%2115453.43%84.92%
IDAdapted99.65%28584.71%85.01%
OODOriginal61.33%2517050.54%82.41%
OODAdapted99.38%410583.20%83.72%

This combined prompt-and-interface adaptation substantially improves output validity, while semantic accuracy among valid outputs changes only marginally. These results suggest that the improvement in overall accuracy is largely associated with increased output validity, although valid-but-incorrect semantic predictions remain.

Overall, the analysis reveals heterogeneous failure profiles among GUI specialists. Their original performance gaps cannot be attributed exclusively to semantic understanding, but semantic errors also persist beyond output-format compliance. We will report these diagnostics and clarify this distinction in the revised paper.

Response to Reviewer 8dxq Q3:

We thank the reviewer for this suggestion. We further decomposed the CoG versus NoCoG+Rule improvements by cognition dimension and held-out platform, using the existing predictions from three matched training seeds (42, 88, and 123). All values below are mean improvements in percentage points (pp), computed over the same evaluation records within each group.

GroupSplitRecords/SeedΔE2E-HitΔStrict Accuracy
Affective valenceID230+0.43+0.29
Source credibilityID219+1.67+2.74
Message framingID120+1.39+7.22
Affective valenceOOD253+1.05+4.35
Source credibilityOOD264+3.54+5.18
Message framingOOD132+6.82+4.55
XOOD231+1.44+6.49
XiaohongshuOOD254+5.38+6.04
WeChat ChannelsOOD164+2.44+0.20

The overall mean improvements are +1.11/+3.24 pp in E2E-Hit and +2.69/+4.73 pp in Strict Accuracy on ID/OOD, respectively.

The disaggregated results show that the aggregate gains are not driven by a single cognition dimension or held-out platform. In particular, OOD Strict Accuracy improves by +4.35, +5.18, and +4.55 pp across affective valence, source credibility, and message framing, respectively.

However, the magnitude of improvement varies across platforms: OOD Strict gains range from +0.20 pp on WeChat Channels to +6.49 pp on X. We therefore interpret these results as positive mean improvements across the evaluated groups, rather than evidence of uniform or statistically significant gains in every setting.

We will include this breakdown in the revised paper to make the distribution and heterogeneity of the improvements explicit.

Reviewer_qmBxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces cognition-conditioned interaction (CCI), a setting in which a GUI agent must first infer a semantic state from social-media content and then ground the action that a fixed policy assigns to that state. The authors construct a 3,916-record benchmark covering affective valence, source credibility, and gain/loss framing, with screenshot-disjoint in-distribution splits and three held-out platforms for OOD evaluation. They train SoGUI-SFT-CoG on Qwen3-VL-8B to emit the cognition label, action, and box in one sequence. Over three seeds, it improves Strict Accuracy over a cognition-free NoCoG baseline by 6.56/12.37 points (ID/OOD) and over a prompt-matched NoCoG+Rule control by 2.69/4.73 points.

Critical strengths
  1. The benchmark enables informative capability diagnosis. Separate annotations for semantic states, policy-derived actions, and target boxes allow the evaluation to distinguish content-understanding errors from grounding failures. The held-out-platform split further tests cross-platform transfer. These analyses show that strong ScreenSpot grounding does not necessarily translate into strong CCI performance and identify challenges in grounding interactions on unfamiliar social interfaces.
  2. The paper is clearly written and well structured. The logic flows fluently from the task formulation and benchmark construction to controlled experiments and capability analysis. The overview figures clearly illustrate the relationship between semantic inference, policy-derived actions, and GUI grounding, while organizing the results around explicit findings makes the empirical argument easy to follow.
Critical weaknesses

The central claim that explicit cognition supervision is a distinct, learnable component is not isolated by the current controls. Each benchmark policy maps cognition labels bijectively to actions, so the action target already determines the cognition label. Adding the cognition field therefore introduces information redundancy. Although CoG outperforms NoCoG+Rule, this comparison cannot distinguish the benefit of explicit semantic supervision from repeated label supervision. The results establish that the cognition-prefixed output format helps, but do not establish why it helps or whether the improvement requires a distinct cognition representation. A matched control using a repeated-action or semantically unrelated prefix is needed to substantiate the interpretation in Finding 3.

Questions for authors
  1. Can you add matched controls that replace the cognition label with a repeated action label or a consistently assigned arbitrary code, while keeping the output structure and training budget comparable? This would clarify whether meaningful semantic labels provide benefits instead of redundant decision supervision.
  2. How does SoGUI-SFT-CoG compare with a modular pipeline in which a frozen MLLM predicts cognition, the fixed policy determines the action, and a GUI-specialist grounder locates the target using the same explicit action instructions as the Oracle-grounding evaluation? This comparison would establish whether joint cognition-conditioned training offers an advantage over composing existing classification and grounding capabilities.
2 comments

Response to Reviewer qmBx W:

We thank the reviewer for identifying this important confound. We agree that, under the current one-to-one cognition-to-action mappings, the original CoG–NoCoG+Rule comparison alone cannot distinguish meaningful semantic supervision from repeated classification supervision or additional prefix effects.

To address this concern, we implemented two matched controls: Repeated-Action, which replaces the cognition label with a repeated action label, and Arbitrary-Code, which replaces it with a consistently assigned, semantically unrelated class code.

All three models use the same Qwen3-VL-8B-Instruct backbone, 2,698 training records, seed 42, three training epochs, and 255 optimizer updates. We preserve the policy instructions, action and box supervision, and output-field order. The Arbitrary-Code control additionally matches the complete target-response token count to CoG on each training record.

Prefix SupervisionID E2E-HitOOD E2E-Hit
Semantic Labels (CoG)94.73%87.98%
Repeated Action93.85%82.59%
Arbitrary Codes94.20%84.28%

On held-out platforms, CoG outperforms Repeated-Action by 5.39 pp and Arbitrary-Code by 3.70 pp, with screenshot-clustered paired-bootstrap 95% confidence intervals of [1.92, 9.54] and [0.96, 6.94] pp, respectively.

These results show that the observed OOD benefit cannot be fully explained by the tested repeated-action or arbitrary-prefix controls. They provide evidence for the practical value of semantically meaningful label supervision, while not establishing that the model learns an independent internal semantic representation.

We will therefore refine Finding 3 as follows:

"In the evaluated setting, explicit semantic-label supervision improves end-to-end interaction on held-out platforms and provides directly inspectable intermediate semantic predictions."

This revised claim reflects the observed supervision effects without making unsupported assertions about the independence of internal representations.

Response to Reviewer qmBx Q1:

Thank you for this constructive suggestion. We implemented both requested controls to examine whether CoG's improvement arises from meaningful semantic-label supervision rather than redundant classification supervision or an additional output prefix.

Specifically, Repeated-Action replaces the cognition label with a repeated action label, while Arbitrary-Code replaces it with a fixed, semantically unrelated class code. Both retain an intermediate decision-bearing prefix before the final action and bounding box.

To ensure comparability, all three models use Qwen3-VL-8B-Instruct, the same 2,698 training records, seed 42, 3 training epochs, and 255 optimizer updates. We preserve the original policy instructions, action and box supervision, and output-field order, changing only the prefix targets and corresponding format instructions. The Arbitrary-Code control additionally matches CoG's complete target-response token count for each training record.

Prefix SupervisionID E2E-HitOOD E2E-Hit
Semantic Labels (CoG, matched rerun)94.73%87.98%
Repeated Action93.85%82.59%
Arbitrary Codes94.20%84.28%

On held-out platforms, CoG outperforms Repeated-Action by 5.39 pp and Arbitrary-Code by 3.70 pp, with screenshot-clustered paired-bootstrap 95% confidence intervals of [1.92, 9.54] and [0.96, 6.94] pp, respectively. These intervals reflect test-sample uncertainty conditional on the seed-42 trained models.

The results indicate that the OOD gains from semantic-label supervision cannot be fully explained by the tested repeated-action or arbitrary-prefix controls. In particular, CoG's advantage over token-matched arbitrary codes supports an additional benefit from meaningful semantic labels beyond simply introducing an intermediate classification target.

We will incorporate these controls and refine Finding 3 to emphasize the observed benefits of semantic-label supervision, without claiming that the experiments establish an independent internal semantic representation.

Response to Reviewer qmBx Q2:

We thank the reviewer for suggesting this important comparison. We implemented the requested modular pipeline: frozen Qwen3-VL-8B-Instruct cognition prediction → predefined policy → frozen GUI-R1 grounding. The grounding stage uses the same explicit action instructions as our Oracle-grounding evaluation. No ground-truth cognition or action labels are used during inference.

Importantly, the modular pipeline uses a separate cognition-only inference pass with the frozen Qwen3-VL-8B-Instruct model, rather than reusing the main-table predictions. This inference protocol requests only the semantic label, with action selection and grounding handled by subsequent components. On OOD, its Cognition Accuracy is 88.75% (576/649), and its E2E-Hit is 87.21% (566/649). Thus, the difference from the main-table cognition scores arises from independent predictions under a different output protocol.

We evaluated the pipeline on the complete ID (569 records) and held-out-platform OOD (649 records) test sets.

MethodID E2E-HitOOD E2E-Hit
SoGUI-SFT-CoG (matched rerun)94.73%87.98%
Frozen Cognition → Policy → GUI-R182.78%87.21%
CoG Cognition → Policy → GUI-R192.09%92.60%

1. Frozen modular comparison. CoG outperforms the frozen pipeline by 11.95 pp on ID, with a screenshot-clustered paired-bootstrap 95% confidence interval of [8.37, 15.24] pp. On OOD, the two systems achieve comparable E2E-Hit (87.98% versus 87.21%).

This comparison evaluates a task-trained CoG model against a pipeline composed of frozen components. We therefore do not interpret it as evidence that joint architectures are inherently superior to modular approaches.

2. Component composition with CoG cognition. To further examine the separability of semantic inference and GUI grounding, we held CoG's predicted cognition labels and policy-derived actions fixed, replacing only its grounding component with GUI-R1.

This combination achieves 92.60% OOD E2E-Hit, outperforming joint CoG by 4.62 pp, with a screenshot-clustered paired-bootstrap 95% confidence interval of [2.75, 6.93] pp. We verified that cognition predictions and action decisions are identical between these two settings.

The three configurations also enable two complementary comparisons:

  • With GUI-R1 grounding fixed, replacing frozen base-model cognition with CoG cognition increases OOD E2E-Hit from 87.21% to 92.60% (+5.39 pp).
  • With CoG cognition and actions fixed, replacing joint CoG grounding with GUI-R1 increases OOD E2E-Hit from 87.98% to 92.60% (+4.62 pp).

These results provide behavioral evidence for the composability of semantic inference and GUI grounding. Explicit semantic predictions can be transferred to an independent GUI specialist, while the choice of grounding component materially affects cross-platform interaction success.

We will incorporate both modular comparisons and clarify their inference protocols, training conditions, and interpretation in the revised paper, without claiming that joint prediction universally outperforms modular alternatives.

Reviewer_Jg4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cognition-conditioned GUI interaction, where an agent first infers the semantic properties of social-media content, then selects and grounds an action under a predefined policy. The authors introduce SoGUI, a benchmark with 3,916 records covering affective valence, source-credibility cues, and gain/loss framing. They fine-tune Qwen3-VL-8B to predict a semantic label followed by the action and bounding box. Through controlled SFT experiments and diagnostic evaluations, the paper argues that explicit cognition supervision improves interaction performance, while localization becomes the main remaining bottleneck.

Critical strengths
  • The benchmark connects content understanding with GUI interaction and provides semantic labels, policy-derived actions, and localization targets. This allows more detailed analysis than reporting only the final success rate.
  • The NoCoG+Rule control is useful. It separates the effect of explicitly providing the policy rule from the effect of supervising the cognition field. The additional Strict Accuracy gains of 2.69/4.73 points on ID/OOD are positive across all three training seeds.
  • The screenshot-disjoint split and evaluation on three held-out platforms are sensible design choices. The action-stratified results also reveal meaningful differences between liking, forwarding, and abstention.
Critical weaknesses
  • The methodological novelty is limited compared with existing intermediate-supervision methods. StaR already supervises current-state perception and desired-state reasoning before deciding whether to act. UI-Ins also supervises an intermediate instruction-reasoning sequence before predicting coordinates, followed by RL refinement. These works already establish the general approach of explicitly supervising an intermediate representation before GUI execution. SoGUI uses a compact social-content label and a fixed label-to-action mapping, but the methodological advance beyond these approaches is not sufficiently clear. The authors may consider adapting state-aware reasoning or instruction-as-reasoning baselines to the same dataset and policy.
  • The novelty of the capability findings should be more clearly distinguished from prior work. SoMe already evaluates social-content understanding, emotion analysis, and whether a user will like, comment on, or repost a post. MMBench-GUI separately evaluates understanding and grounding, and its controlled planner/grounder comparisons identify grounding as a major bottleneck. SoGUI adds a distinguishable benchmark connecting annotated social semantics to screenshot localization under controlled policies. However, the broader observations that content understanding matters and that understanding and grounding have different limitations are incremental. What new insight is obtained specifically from this social-semantic setting? A semantic classifier followed by a deterministic policy and an existing GUI grounder would also be a natural baseline for evaluating this contribution.
  • The benefit of cognition supervision is not fully isolated from redundant classification supervision. To my understanding, the cognition-to-action mapping in Table 15 is one-to-one within each task. Given the policy, the action target in NoCoG+Rule already contains the same classification information as the cognition label. CoG adds another supervised prediction of this information before generating the action and box. What if the model is trained to repeat the action, or output an arbitrary class ID, before predicting the location? Without such a control, it is unclear whether the improvement comes from the semantic meaning of the cognition field or from additional prefix supervision and loss weighting. Appendix D acknowledges this limitation, but it is central to interpreting the main result.
  • The conclusion that localization becomes the dominant bottleneck depends heavily on the box-quality criterion. Table 3 uses Box-Strict, although E2E-Hit is presented as the primary interaction metric. On OOD, cognition errors account for 6.16% of examples, while cognition-correct E2E failures account for approximately 93.84 − 87.21 = 6.63%. These two error rates are close. The much larger 28.51% localization failure rate in Table 3 includes many predictions that already select the correct action and click inside the correct control, but fail the IoU threshold. The authors should report the decomposition under E2E-Hit and explain why precise box extent is necessary for the intended interaction.
  • There is an inconsistency between the Oracle diagnostic and the stated metric definitions. Appendix I.3 defines Cog-Hit as a weaker criterion than E2E-Hit: it removes action correctness on actionable examples and accepts either the no-action label or the sentinel on non-actionable examples. For the same predictions, Cog-Hit should therefore be at least as high as E2E-Hit. However, the normal ID scores in Table 19 are 65.03 for GUI-R1 and 45.17 for POINTS, below their E2E-Hit scores of 69.77 and 58.52 in Table 1. Are these different normal runs or different scoring protocols? This should be clarified before using the Oracle gains as evidence for the capability decomposition.
Questions for authors

See weaknesses

4 comments

Response to Reviewer Jg4U W1:

We thank the reviewer for highlighting the connections to StaR and UI-Ins. We agree that intermediate supervision itself is not new. However, as clarified in our Introduction and Figure 1, SoGUI's contribution extends beyond introducing another intermediate-supervision method.

1. A distinct capability and diagnostic formulation. Existing GUI research has primarily studied goal-conditioned grounding and state-aware execution, where the intended goal or action is externally specified. In contrast, CCI requires agents to infer task-relevant semantic states from observed content before determining and grounding an interaction under a specified policy. SoGUI formalizes this capability and provides separate annotations for semantic inference, policy-derived actions, and GUI grounding, enabling controlled failure diagnosis.

2. Task-relevant semantic supervision. Unlike intermediate descriptions centered on interface states or grounding instructions, CoG explicitly supervises an observable semantic state that connects content interpretation to policy-conditioned interaction. This compact intermediate field is directly annotated and independently evaluable, without requiring full natural-language reasoning traces.

3. Additional controlled comparison. Following the reviewer's suggestion, we implemented a UI-Ins-inspired SFT baseline that generates a brief description of the target control's function, position, or appearance before predicting the action and box. Both models use Qwen3-VL-8B, the same 2,698 training records, policies, LoRA configuration, and matched training budget (3 epochs, 255 updates, seed 42).

MethodID E2E-HitOOD E2E-HitID TokensOOD Tokens
UI-Ins-inspired SFT93.85%86.44%39.6040.16
CoG94.73%87.98%32.8532.86

CoG achieves slightly higher E2E-Hit (+0.88/+1.54 pp on ID/OOD) with approximately 17%–18% shorter outputs. Complementary prefix controls further show OOD improvements of 5.39 pp over Repeated-Action and 3.70 pp over Arbitrary-Code, providing evidence that meaningful semantic-label supervision offers benefits beyond the tested redundant and arbitrary intermediate targets.

In summary, our contribution is not a new intermediate-supervision paradigm, but the formulation and diagnostic evaluation of content-conditioned GUI interaction, together with controlled evidence that compact, task-relevant semantic supervision can improve interaction performance. We will clarify this distinction from prior intermediate-supervision approaches in the revised paper.

Response to Reviewer Jg4U W2:

We thank the reviewer for highlighting the connections to SoMe and MMBench-GUI. We agree that social-content understanding and the distinction between GUI understanding and grounding have been studied previously. SoMe evaluates social-content understanding and interaction-related predictions, while MMBench-GUI separately evaluates GUI understanding and grounding, including planner/grounder comparisons.

SoGUI's specific contribution is an explicit, controllable policy layer connecting social-semantic states, policy-derived actions, and GUI grounding targets within the same task records. This enables controlled examination of how semantic judgments, interaction policies, and grounding components jointly affect interaction success.

Our additional experiments reveal two concrete properties of this setting.

1. Semantic judgments can be combined with changed interaction policies. We evaluated the original seed-42 CoG and NoCoG+Rule models under changed test-time policies without further training. We reversed the emotion and framing mappings and cyclically permuted the credibility mapping while preserving screenshots and ground-truth semantic labels.

ModelID Action Acc. (Changed Policy)OOD Action Acc. (Changed Policy)
NoCoG+Rule80.14%83.51%
CoG93.67%89.83%
Improvement+13.53 pp+6.32 pp

The screenshot-clustered paired-bootstrap 95% confidence intervals are [9.87, 17.43] pp (ID) and [3.30, 9.23] pp (OOD). Moreover, CoG's emitted actions are 100% consistent with applying the changed policies to its predicted cognition labels. These results provide evidence that explicit semantic supervision supports adaptation to the tested policy changes beyond the original fixed mappings.

2. Semantic inference and GUI grounding are behaviorally composable. We implemented the suggested modular pipeline using frozen Qwen3-VL-8B-Instruct for cognition prediction, the predefined interaction policy, and frozen GUI-R1 for grounding. The cognition stage uses an independent cognition-only inference protocol rather than reusing main-table predictions. Its OOD Cognition Accuracy is 88.75% (576/649), consistent with its 87.21% (566/649) E2E-Hit.

We further held CoG's predicted cognition labels and actions fixed, replacing only its grounding component with GUI-R1.

MethodID E2E-HitOOD E2E-Hit
Frozen Cognition + Policy + GUI-R182.78%87.21%
Joint CoG (matched rerun)94.73%87.98%
CoG Cognition + Policy + GUI-R192.09%92.60%

With CoG cognition and actions held fixed, replacing its grounder with GUI-R1 reduces ID E2E-Hit by 2.64 pp but improves OOD E2E-Hit by 4.62 pp, with a screenshot-clustered paired-bootstrap 95% confidence interval of [2.75, 6.93] pp for the OOD improvement.

This reveals a concrete cross-platform difference: identical semantic decisions yield different interaction outcomes depending on the grounding component and platform distribution. It also demonstrates that explicit semantic predictions can be transferred to an independent GUI grounder.

In summary, SoGUI goes beyond separately evaluating social semantics or GUI grounding by enabling controlled examination of their interaction through explicit policies. The additional experiments provide behavioral evidence for the compositionality of semantic inference, interaction policies, and GUI grounding. We will revise our findings to emphasize these measurable properties rather than claim novelty in social-content understanding or understanding–grounding separation alone.

Response to Reviewer Jg4U W3:

We thank the reviewer for identifying this important confound. We agree that, under the one-to-one cognition-to-action mappings, the original CoG–NoCoG+Rule comparison alone cannot distinguish semantic-label supervision from redundant classification or additional prefix supervision.

To address this concern, we implemented two matched controls: Repeated-Action, which replaces the cognition label with a repeated action label, and Arbitrary-Code, which replaces it with a fixed, semantically unrelated class code.

All three conditions use Qwen3-VL-8B-Instruct, the same 2,698 training records, policy prompts, training budget (3 epochs, 255 optimizer updates), and seed 42. CoG was retrained alongside the two controls under this matched configuration. The Arbitrary-Code control additionally matches CoG's complete target-response token count for each training example.

Prefix SupervisionID E2E-HitOOD E2E-Hit
Semantic Labels (CoG, matched rerun)94.73%87.98%
Repeated-Action93.85%82.59%
Arbitrary-Code94.20%84.28%

On held-out platforms, CoG outperforms Repeated-Action by 5.39 pp and Arbitrary-Code by 3.70 pp, with screenshot-clustered paired-bootstrap 95% confidence intervals of [1.92, 9.54] and [0.96, 6.94] pp, respectively. These intervals reflect test-sample uncertainty conditional on the seed-42 trained models.

These results indicate that CoG's OOD gains cannot be fully explained by the tested repeated-action or arbitrary-prefix controls. They provide evidence for the empirical benefit of meaningful semantic-label supervision beyond redundant decision targets.

We will therefore refine Finding 3 as follows:

“In the evaluated setting, explicit semantic-label supervision improves end-to-end interaction on held-out platforms and provides directly inspectable intermediate semantic predictions.”

This revised claim reflects the observed supervision benefits without asserting that the experiments establish an independent internal semantic representation.

Response to Reviewer Jg4U W4:

We thank the reviewer for identifying this important distinction. We agree that the original Finding 4 conflated click-level interaction failures with insufficient bounding-box overlap.

We reanalyzed the original CoG predictions and separated cognition errors, missed clicks with correct cognition, and successful clicks whose predicted boxes fail the IoU ≥ 0.65 requirement.

Failure CategoryIDOOD
Incorrect cognition2.81%6.16%
Correct cognition, failed click3.16%6.63%
Successful click, IoU < 0.658.26%21.88%
Box-Strict success85.76%65.33%

The corresponding E2E-Hit scores are 94.02%/87.21% on ID/OOD. As the reviewer noted, cognition errors and missed clicks contribute comparable shares of end-to-end failures (2.81% vs. 3.16% on ID; 6.16% vs. 6.63% on OOD). Most additional Box-Strict failures arise from successful clicks with insufficient box overlap, especially on OOD.

For the single-step interaction studied here, accurate prediction of the entire bounding-box extent is not necessary for a successful click. E2E-Hit remains the primary interaction metric, while Box-Strict is retained as a supplementary diagnostic of predicted-region quality, not a requirement for interaction success.

We will therefore revise Finding 4 as follows:

"Under click-level E2E-Hit, semantic errors and missed clicks contribute comparable shares of residual failures. Under Box-Strict, insufficient box overlap accounts for the largest additional failure component, particularly on held-out platforms."

We will align the abstract, Table 3 discussion, GRPO motivation, and conclusion with this metric-specific interpretation, without claiming that missed clicks dominate the remaining interaction failures.

Response to Reviewer Jg4U W5:

We thank the reviewer for identifying this apparent inconsistency. We agree that, for identical predictions, Cog-Hit must be at least as high as E2E-Hit.

1. Clarification of evaluation protocols. Table 1 and Table 19 use independent prediction sets obtained under different prompting protocols. Table 1 reports the main evaluation, whereas Table 19 is a separate paired Normal/Oracle diagnostic. Within each pair, the images, model configurations, decoding settings, and output handling are matched; the Oracle condition additionally provides the ground-truth cognition label. The Normal predictions in Table 19 are therefore not identical to those used in Table 1.

2. Verification of metric consistency. To verify the scoring implementation, we recomputed E2E-Hit and Cog-Hit on the same original Table 1 predictions, using the original parsers and coordinate conversions.

ModelSplitE2E-HitCog-Hit
GUI-R1ID69.77%69.95%
GUI-R1OOD62.25%62.40%
POINTSID58.52%64.67%
POINTSOOD43.76%49.00%

All 2,436 individual predictions satisfy the expected relationship, with zero cases where E2E-Hit succeeds but Cog-Hit fails. Thus, the apparent inconsistency results from comparing metrics computed on different prediction sets, rather than a violation of the metric definitions.

3. Oracle diagnostic interpretation. We also verified the paired Normal/Oracle results within Table 19. The original Cog-Hit improvements are reproducible: +17.22/+11.09 pp for GUI-R1 and +15.82/+18.34 pp for POINTS on ID/OOD. These gains are calculated relative to the matched Normal condition within Table 19 and quantify the effect of providing correct semantic information under that diagnostic protocol.

We will clarify the independent evaluation protocols in the Table 19 caption and Appendix I.3, document the relevant prompting and scoring conventions, and include the same-prediction metric consistency check.

Reviewer_BKQJOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SoGUI studies offline social-media GUI interaction under fixed, synthetic task policies. Depending on the task, the model predicts affective valence, source credibility from visible cues, or gain/loss framing; it then derives a like, forward, or no-action decision from the policy and, when an action is required, localizes the corresponding control. The paper introduces a benchmark of 3,916 annotated records and CoG, a Qwen3-VL-8B model fine-tuned to generate an explicit semantic label before the action and bounding box. Evaluation uses a screenshot-disjoint ID split and a held-out-platform OOD split. The experiments also include prompt-matched supervised fine-tuning controls, an Oracle-cognition diagnostic, and a preliminary GRPO refinement.

Critical strengths
  1. The NoCoG+Rule control shares the pipeline with CoG but omits the semantic output field. CoG improves Box-Strict in all tested settings. This comparison separates the benefit of supervising the semantic field from the benefit of supplying the policy rules.

  2. The benchmark allows semantic inference and action-specific grounding to be evaluated separately. Records that share a screenshot are kept in the same split, and three platforms are held out for evaluation. E2E-Hit provides a common action-and-point metric with rules for no-action predictions and parsing failures. The documented denominators and per-action breakdowns make it possible to trace where the aggregate scores come from.

Critical weaknesses
  1. The scope is narrow, and the paper does not show that the setting matters beyond this benchmark.

    • Each task uses one fixed rule, stated in the prompt (for example, positive means like, negative means no action), and the authors note that these rules are synthetic.
    • The task then amounts to a two- or three-way classification of the post, plus a click on the Like or Share button when needed. Without reported fine-tuning on this benchmark, GPT-5.6 Luna already hits the target as often as CoG (Center Hit) and is within six E2E-Hit points of it.
    • Because training and testing use the same rule, the results cannot tell whether a model follows the stated rule or has learned a fixed mapping from content to action.
  2. The paper attributes the GUI specialists' weaker results to semantic inference (Findings 1 and 2), but the specialists were not trained on this benchmark, so the results do not separate this from unfamiliarity with the task and its output format.

    • The specialists are prompted to give a label and an action in the task's format, and malformed outputs are not repaired. In Case A, POINTS-GUI-G answers "like or no-action", which is scored as a failed action. The paper does not report how often this happens.
  3. Finding 4's localization bottleneck mostly reflects box precision, not missed clicks.

    • Table 3 counts a correct-cognition example as a localization failure whenever it fails Box-Strict, which also requires IoU ≥ 0.65 with the annotated box.
    • On actionable examples, CoG's E2E-Hit requires the correct action and a predicted box center inside the target, with no IoU threshold. Most of Table 3's correct-cognition failures are successful clicks with IoU below 0.65.
    • Because CoG's actions follow its labels, as the paper states, the correct-cognition failures left under E2E-Hit are about as frequent as cognition errors. Localization is therefore not clearly the dominant bottleneck.
Questions for authors
  1. Evaluate CoG and NoCoG+Rule with a changed rule at test time (for example, negative-like) and report action accuracy. This would test whether the models can follow a new stated rule.

  2. Fine-tune one GUI specialist, such as GUI-R1-7B, on the same training data and report its E2E-Hit on ID and OOD. This would test how much task-specific training closes the observed gap.

  3. Report how often each GUI specialist's output does not give a single valid label or action, as in Case A.

3 comments

Response to Reviewer BKQJ W1 &Q1:

We thank the reviewer for raising these important concerns. We clarify the motivation for the controlled benchmark and provide additional evidence regarding generalization beyond the original fixed policies.

1. Controlled benchmark and task scope. The purpose of SoGUI is not to maximize task complexity, but to provide a reproducible and diagnostic evaluation of content-conditioned GUI interaction. Unlike conventional GUI grounding tasks that typically specify the intended action, CCI requires agents to infer task-relevant semantic states from observed content and translate them into GUI interactions under a specified policy.

Affective valence, visible source credibility, and message framing are theoretically grounded semantic constructs. Their predefined labels and deterministic mappings provide controlled instances of CCI rather than an exhaustive representation of social interaction. By separately annotating semantic states, policy-derived actions, and GUI targets, SoGUI enables diagnosis of semantic inference, policy-consistent action selection, and GUI grounding. The fixed policies therefore serve as experimental controls, not assumptions about real-world social decision-making.

2. Generalization beyond fixed mappings (Q1). To examine whether models merely memorize the original content-to-action mappings, we evaluated the seed-42 CoG and NoCoG+Rule models under changed test-time policies without further fine-tuning. We reversed the emotion and framing mappings and cyclically permuted the credibility mapping, keeping screenshots and ground-truth semantic labels unchanged.

ModelID OriginalID ChangedOOD OriginalOOD Changed
CoG96.84%93.67%93.68%89.83%
NoCoG+Rule96.31%80.14%91.83%83.51%
CoG − NoCoG+Rule+0.53 pp+13.53 pp+1.85 pp+6.32 pp

All values are Action Accuracy. The Original columns report original-policy controls from the new paired evaluation runs, rather than the original Table 1 results. The Original and Changed conditions use the same seed-42 checkpoints and matched evaluation settings within this experiment.

Under the changed policies, CoG outperforms NoCoG+Rule by 13.53/6.32 pp on ID/OOD, with screenshot-clustered paired-bootstrap 95% confidence intervals of [9.87, 17.43]/[3.30, 9.23] pp.

Moreover, CoG's emitted actions are 100% consistent with applying the changed policies to its predicted cognition labels. These results provide evidence that explicit cognition supervision improves adaptation to the tested policy changes, beyond its benefits under the original fixed mappings. Since some actions introduced by the changed policies lack corresponding annotated target boxes, we report Action Accuracy rather than E2E-Hit for this experiment.

3. Task simplicity and GPT-5.6 Luna. We agree that Luna's competitive zero-shot performance demonstrates that the current tasks are accessible to strong general-purpose MLLMs. However, a controlled diagnostic benchmark need not maximize task difficulty to provide meaningful capability measurements. Its value lies in distinguishing semantic inference from GUI grounding and identifying their respective contributions to interaction success.

Our claim is not that CoG universally outperforms stronger general-purpose models. Rather, the matched CoG–NoCoG+Rule comparison provides controlled evidence for the benefit of cognition-target supervision. Using the same Qwen3-VL-8B backbone, training data, policy instructions, and optimization settings, CoG improves Strict Accuracy by 2.69/4.73 pp on ID/OOD across three seeds.

In summary, SoGUI does not aim to reproduce the full complexity of real-world social decision-making. It establishes a controlled evaluation of a prerequisite capability for content-conditioned GUI interaction and provides empirical evidence for the benefits of explicit semantic supervision. We will clarify this scope, report the policy-change results, and temper claims regarding task difficulty and broader generalization in the revised paper.

Response to Reviewer BKQJ W2 &Q2&Q3:

We thank the reviewer for highlighting the potential confounding effects of output protocols and task familiarity. We agree that specialists' original performance gaps should not be attributed exclusively to semantic understanding. We conducted an output audit and a staged adaptation experiment to distinguish these factors.

1. Original output audit . Although the original prompts specified task rules, allowed labels, output formats, and coordinate examples, some specialists produced invalid outputs. We audited all original predictions (569 ID, 649 OOD records).

ModelInvalid Cognition (ID/OOD)Invalid Action (ID/OOD)Either Invalid (ID/OOD)
UI-R1-E37.08% / 38.67%23.55% / 22.19%37.08% / 38.67%
GUI-R1-7B7.21% / 4.16%0.88% / 0.77%7.21% / 4.16%
POINTS-GUI-G17.75% / 19.88%14.94% / 10.94%17.75% / 19.88%
ScaleCUA-7B21.44% / 19.26%15.99% / 18.80%21.44% / 19.41%

Invalid outputs include missing fields, copied alternatives, or labels outside the allowed set; "Either Invalid" denotes the union of invalid cognition and action outputs.

For POINTS-GUI-G (Figure 6), all 85 ID and 71 OOD invalid action outputs contain copied alternatives. Among valid cognition outputs, 102/468 ID and 120/520 OOD predictions remain semantically incorrect. Thus, both protocol violations and semantic errors contribute to the observed failures.

2. Staged protocol adaptation and fine-tuning. We selected UI-R1-E-3B because it had the lowest original joint cognition/action validity (62.92%/61.33% on ID/OOD), making it informative for separating protocol unfamiliarity from task adaptation.

First, we clarified the instructions and adopted its native pixel-point protocol without parameter updates. We then fine-tuned it on the same 2,698 training records (3 epochs, 255 updates, seed 42), using cognition labels, policy-derived actions, and annotated box-center points. The adapted protocol and scoring remained fixed across the before/after SFT comparison.

MetricID BeforeID AfterOOD BeforeOOD After
Valid Cognition + Action99.65%100%99.38%100%
Cognition Acc.84.71%95.78%83.20%93.99%
Action Acc.81.55%95.78%79.35%93.99%
Center Hit64.99%85.15%48.20%62.89%
E2E-Hit49.38%89.28%35.13%73.65%

Before SFT denotes protocol-adapted evaluation without fine-tuning. Valid Cognition + Action measures label validity, not coordinate validity.

Protocol adaptation increased joint output validity from 62.92%/61.33% to 99.65%/99.38%. With the protocol fixed, SFT further improved Cognition Accuracy by 11.07/10.79 pp and E2E-Hit by approximately 39.9/38.5 pp on ID/OOD.

Relative to the reference SoGUI-SFT-CoG results, UI-R1-E's E2E-Hit gap decreased from 44.64/52.08 pp before SFT to 4.74/13.56 pp afterward. This descriptive comparison involves different backbones and output protocols, but demonstrates that task-specific training substantially narrows the observed gap.

3. Implications for Findings 1–2. These results show that original specialist performance reflects both protocol adherence and task familiarity, alongside semantic inference and grounding capabilities. We will revise Findings 1–2 to avoid interpreting zero-shot gaps as intrinsic semantic deficiencies of GUI specialists. The benefit of explicit cognition-target supervision is assessed separately through matched CoG–NoCoG+Rule comparisons under the same backbone, training data, and optimization settings.

**Response to Reviewer BKQJ W3: **

We thank the reviewer for identifying this important distinction. We agree that the original Finding 4 conflated click-level interaction failures with errors in box localization precision.

We reanalyzed the original CoG predictions and separated cognition errors, missed clicks with correct cognition, and successful clicks whose predicted boxes fail the IoU ≥ 0.65 requirement.

Failure CategoryIDOOD
Incorrect cognition2.81%6.16%
Correct cognition, missed click3.16%6.63%
Correct cognition, successful click but IoU < 0.658.26%21.88%
Total Box-Strict failure14.24%34.67%

The categories are mutually exclusive. Minor discrepancies in sums arise from rounding.

The results confirm the reviewer's observation. Most cognition-correct Box-Strict failures are successful clicks with insufficient box overlap, particularly on OOD (21.88% versus 6.63% missed clicks). Under E2E-Hit, cognition errors and missed clicks contribute comparable shares (2.81% versus 3.16% on ID; 6.16% versus 6.63% on OOD).

We will therefore revise Finding 4 to distinguish these metrics:

"After cognition improves, insufficient box overlap accounts for the largest share of residual Box-Strict failures, particularly on held-out platforms; under click-level E2E-Hit, semantic errors and missed clicks contribute comparable shares."

We will align the abstract, Table 3 discussion, GRPO motivation, and conclusion with this metric-specific interpretation, without claiming that missed clicks dominate residual interaction failures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.