- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper investigates semantic-state inference and GUI grounding in social GUI agents, arguing that these are partially separable capabilities. It introduces a benchmark with 3,916 records from six platforms, covering affective valence, source-credibility cues, and message framing. For each task, the agent must infer the semantic state, determine the action under a predefined policy, and locate the corresponding GUI element when an interaction is required. The paper evaluates GUI-specialized models, general-purpose VLMs, and its SoGUI models. The reference SoGUI model achieves 94.02% E2E-Hit on ID evaluation and 87.21% on held-out platforms. Explicit semantic-state supervision improves Strict Accuracy over the prompt-matched baseline by 2.69 and 4.73 percentage points, respectively. Finally, the authors apply GRPO to address the remaining localization errors.
Critical strengths
1- The paper studies a well-motivated problem. GUI agents need to understand the content before deciding whether and how to interact. The benchmark provides separate annotations for semantic states, policy-derived actions, and GUI locations, making it possible to examine where interaction failures occur. This is a useful contribution toward understanding capabilities that conventional grounding accuracy alone does not capture.
2- The controlled supervision comparison provides useful evidence. The CoG and NoCoG+Rule conditions use the same training data and policy instructions, allowing the authors to examine the benefit of explicitly supervising the semantic state. The reported Strict Accuracy improvements are consistent across three training seeds, averaging 2.69 percentage points on ID and 4.73 points on held-out platforms. This strengthens the evidence that explicit semantic supervision can improve cognition-conditioned interaction.
Critical weaknesses
1- The benchmark substantially simplifies the semantic understanding required in real social interactions. Real social content can involve mixed emotions, uncertain credibility, and context-dependent meanings, with several factors jointly influencing an interaction decision. The benchmark represents these judgments through seven predefined labels across three separate tasks, each mapped to a fixed action. This makes it unclear how well the reported improvements extend to situations where semantic states are ambiguous or cannot be captured by these categories. Consequently, the broader significance of the contribution for realistic social GUI interaction remains insufficiently established.
2- The model comparisons may confound semantic understanding with instruction following. In Figure 6, POINTS-GUI-G outputs the alternatives “gain or loss” and “like or no-action” instead of selecting valid labels. This raises the possibility that some failures of specialized GUI agents reflect instruction-following or output-format difficulties. Without quantifying these failures separately, it is difficult to attribute their lower scores specifically to weaker semantic understanding. This makes the interpretation of the cross-model capability comparisons less conclusive.
Questions for authors
1- Dataset clarification: How were posts with mixed emotions, uncertain credibility, or ambiguous framing handled during dataset construction? Were these cases excluded or assigned a single label, and approximately what proportion of the collected content did they represent? This would help clarify how much semantic complexity the benchmark captures.
2- Evaluation clarification: For the GUI-specialized agents, how frequently do failures involve invalid or copied output labels, as illustrated in Figure 6? Could you separate these failures from valid but incorrect semantic predictions, and report semantic accuracy on valid outputs alongside the original scores? This would help assess how much instruction-following difficulties affect the capability comparison.
3- Results clarification: Could you report the CoG versus NoCoG+Rule improvements in E2E-Hit and Strict Accuracy separately for each cognition dimension and held-out platform? This would clarify whether the aggregate gains reflect consistent improvements or are concentrated in particular tasks or interfaces.
5 comments
Response to Reviewer 8dxq W1:
We thank the reviewer for identifying this potential confound. We agree that instruction-following and output-format failures should not be conflated with semantic understanding errors. We conducted additional analyses to separate these failure sources.
1. Separating invalid outputs from semantic errors. We audited the original predictions and distinguished invalid cognition outputs (e.g., copied alternatives or missing labels) from valid but incorrect semantic predictions.
Table 1. Original prediction error decomposition.
| Model | Split | Invalid Cognition | Valid but Incorrect | Accuracy on Valid Outputs |
|---|---|---|---|---|
| UI-R1-E | ID | 37.08% | 54 | 84.92% |
| UI-R1-E | OOD | 38.67% | 70 | 82.41% |
| GUI-R1-7B | ID | 7.21% | 59 | 88.83% |
| GUI-R1-7B | OOD | 4.16% | 110 | 82.32% |
| POINTS-GUI-G | ID | 17.75% | 102 | 78.21% |
| POINTS-GUI-G | OOD | 19.88% | 120 | 76.92% |
| ScaleCUA-7B | ID | 21.44% | 43 | 90.38% |
| ScaleCUA-7B | OOD | 19.26% | 75 | 85.69% |
For POINTS-GUI-G, all invalid cognition outputs contain unselected alternatives, as illustrated in Figure 6. However, semantic errors remain among valid outputs, confirming that both failure types contribute to performance.
2. Prompt-and-protocol adaptation. We additionally evaluated UI-R1-E using clarified task instructions and a native action-coordinate interface, without task-specific fine-tuning. This combined adaptation tests protocol sensitivity without isolating individual modifications.
Table 2. UI-R1-E before and after protocol adaptation.
| Metric | Original ID | Adapted ID | Original OOD | Adapted OOD |
|---|---|---|---|---|
| Valid cognition outputs | 62.92% | 99.65% | 61.33% | 99.38% |
| Overall semantic accuracy | 53.43% | 84.71% | 50.54% | 83.20% |
| Accuracy on valid outputs | 84.92% | 85.01% | 82.41% | 83.72% |
The adaptation substantially increases output validity and overall semantic accuracy, while accuracy among valid outputs changes only marginally. This suggests that the overall improvement is largely associated with increased output validity, although valid-but-incorrect semantic predictions remain.
Together, these results reveal distinct failure sources: output-protocol violations and incorrect semantic predictions. We agree that the specialists' lower end-to-end performance should not be attributed exclusively to semantic understanding. We will revise Findings 1 and 2 accordingly, distinguishing original-protocol performance from semantic accuracy on valid outputs and explicitly acknowledging instruction-following and protocol sensitivity.
Response to Reviewer 8dxq Q1:
Thank you for this question. Posts that did not admit a single defensible label under the task-specific annotation criteria were excluded rather than force-labeled. We retained cases with sufficiently observable evidence in the screenshot and accompanying text, without relying on external context. Clearly mixed affective signals, insufficient source-credibility cues, and indeterminate gain/loss framing did not meet the inclusion criteria.
For retained samples, annotators assigned task-specific semantic labels using predefined criteria: affective language and visual cues for valence, visible account identity and verification cues for credibility, and gain/loss emphasis for framing. Annotation disagreements were resolved through adjudication before finalizing the labels.
We conducted an independent annotation-quality audit with three annotators on 500 class-balanced samples per dimension:
| Dimension | Samples | Unanimous Agreement | Fleiss' κ |
|---|---|---|---|
| Affective valence | 500 | 91.4% | 0.886 |
| Source credibility | 500 | 84.2% | 0.833 |
| Message framing | 500 | 81.4% | 0.764 |
These results assess label reliability within the retained benchmark, not the prevalence of semantic ambiguity in the original collection. The available construction records do not preserve a verified ambiguity-specific exclusion count, so we cannot reliably estimate the proportion of excluded ambiguous posts.
Accordingly, the benchmark focuses on controlled, single-label cases supported by observable task-relevant evidence, rather than representing the full distribution of social-media content or evaluating ambiguous semantic judgments. This controlled scope enables reproducible evaluation and clearer attribution of semantic inference and GUI grounding errors. We will clarify the inclusion criteria, adjudication procedure, and scope limitations in the revised paper.
Response to Reviewer 8dxq Q2:
We thank the reviewer for this suggestion. We audited the original main-table predictions using the original semantic-field parsing rules, distinguishing invalid cognition outputs (e.g., copied alternatives, missing labels, or labels outside the allowed set) from valid but incorrect semantic predictions. ID and OOD contain 569 and 649 records, respectively.
| Model | Split | Invalid Cognition, n (%) | Valid but Incorrect, n | Overall Semantic Acc. | Valid-Output Acc. |
|---|---|---|---|---|---|
| UI-R1-E | ID | 211 (37.08%) | 54 | 53.43% | 84.92% |
| UI-R1-E | OOD | 251 (38.67%) | 70 | 50.54% | 82.41% |
| GUI-R1-7B | ID | 41 (7.21%) | 59 | 82.43% | 88.83% |
| GUI-R1-7B | OOD | 27 (4.16%) | 110 | 78.89% | 82.32% |
| POINTS-GUI-G | ID | 101 (17.75%) | 102 | 64.32% | 78.21% |
| POINTS-GUI-G | OOD | 129 (19.88%) | 120 | 61.63% | 76.92% |
| ScaleCUA-7B | ID | 122 (21.44%) | 43 | 71.00% | 90.38% |
| ScaleCUA-7B | OOD | 125 (19.26%) | 75 | 69.18% | 85.69% |
Overall semantic accuracy uses all records as the denominator, whereas valid-output accuracy is conditional on a single allowed cognition label being extracted.
For POINTS-GUI-G, all 101 ID and 129 OOD invalid cognition outputs contain unselected alternatives, consistent with Figure 6. Its invalid action outputs occur in 85/569 ID cases (14.94%) and 71/649 OOD cases (10.94%). Thus, output-protocol failures and valid-but-incorrect semantic predictions are both substantial.
We additionally evaluated UI-R1-E with clarified instructions and a native action-coordinate interface, without task-specific fine-tuning:
| Split | Setting | Valid Cognition | Invalid, n | Valid but Incorrect, n | Overall Semantic Acc. | Valid-Output Acc. |
|---|---|---|---|---|---|---|
| ID | Original | 62.92% | 211 | 54 | 53.43% | 84.92% |
| ID | Adapted | 99.65% | 2 | 85 | 84.71% | 85.01% |
| OOD | Original | 61.33% | 251 | 70 | 50.54% | 82.41% |
| OOD | Adapted | 99.38% | 4 | 105 | 83.20% | 83.72% |
This combined prompt-and-interface adaptation substantially improves output validity, while semantic accuracy among valid outputs changes only marginally. These results suggest that the improvement in overall accuracy is largely associated with increased output validity, although valid-but-incorrect semantic predictions remain.
Overall, the analysis reveals heterogeneous failure profiles among GUI specialists. Their original performance gaps cannot be attributed exclusively to semantic understanding, but semantic errors also persist beyond output-format compliance. We will report these diagnostics and clarify this distinction in the revised paper.
Response to Reviewer 8dxq Q3:
We thank the reviewer for this suggestion. We further decomposed the CoG versus NoCoG+Rule improvements by cognition dimension and held-out platform, using the existing predictions from three matched training seeds (42, 88, and 123). All values below are mean improvements in percentage points (pp), computed over the same evaluation records within each group.
| Group | Split | Records/Seed | ΔE2E-Hit | ΔStrict Accuracy |
|---|---|---|---|---|
| Affective valence | ID | 230 | +0.43 | +0.29 |
| Source credibility | ID | 219 | +1.67 | +2.74 |
| Message framing | ID | 120 | +1.39 | +7.22 |
| Affective valence | OOD | 253 | +1.05 | +4.35 |
| Source credibility | OOD | 264 | +3.54 | +5.18 |
| Message framing | OOD | 132 | +6.82 | +4.55 |
| X | OOD | 231 | +1.44 | +6.49 |
| Xiaohongshu | OOD | 254 | +5.38 | +6.04 |
| WeChat Channels | OOD | 164 | +2.44 | +0.20 |
The overall mean improvements are +1.11/+3.24 pp in E2E-Hit and +2.69/+4.73 pp in Strict Accuracy on ID/OOD, respectively.
The disaggregated results show that the aggregate gains are not driven by a single cognition dimension or held-out platform. In particular, OOD Strict Accuracy improves by +4.35, +5.18, and +4.55 pp across affective valence, source credibility, and message framing, respectively.
However, the magnitude of improvement varies across platforms: OOD Strict gains range from +0.20 pp on WeChat Channels to +6.49 pp on X. We therefore interpret these results as positive mean improvements across the evaluated groups, rather than evidence of uniform or statistically significant gains in every setting.
We will include this breakdown in the revised paper to make the distribution and heterogeneity of the improvements explicit.
Response to Reviewer 8dxq W1:
We thank the reviewer for this important concern. We agree that real-world social interactions involve semantic ambiguity, mixed emotions, and multiple interacting factors. We would like to clarify the distinction between the general CCI capability and its controlled benchmark instantiation.
1. Rationale for the controlled design. SoGUI evaluates whether GUI agents can infer interaction-relevant semantic states from observed content and execute a specified interaction policy. Affective valence, visible source credibility, and message framing are theoretically grounded, interaction-relevant semantic constructs, rather than an exhaustive taxonomy of social semantics. These dimensions and deterministic mappings instantiate CCI under controlled conditions; they do not define the limits of the general formulation.
This design enables separate diagnosis of semantic inference, policy-consistent action selection, and GUI grounding. Thus, the benchmark serves as a diagnostic instrument for content-conditioned interaction, rather than a comprehensive simulation of real-world social decision-making.
2. Generalization beyond fixed policies. To examine whether models merely memorize the original semantic-to-action mappings, we evaluated the seed-42 CoG and NoCoG+Rule models under changed test-time policies without further fine-tuning. We reversed the emotion and framing mappings and cyclically permuted the credibility mapping, keeping screenshots and ground-truth semantic labels unchanged.
The screenshot-clustered paired-bootstrap 95% confidence intervals are [9.87, 17.43] pp (ID) and [3.30, 9.23] pp (OOD). These results provide evidence that the benefit of explicit semantic supervision extends beyond the original fixed action mappings.
3. Scope and limitations. We acknowledge that the current benchmark does not cover ambiguous semantic states or multi-factor social decisions. Our claim is not that SoGUI captures the full complexity of social interaction, but that it establishes a reproducible diagnostic evaluation of a fundamental capability for content-conditioned GUI interaction: inferring task-relevant semantic states from observed content to guide interaction. We will clarify the distinction between the general CCI formulation and its controlled benchmark instantiation, and explicitly discuss semantic ambiguity and multi-factor decisions as limitations and future extensions.