acceptodds
Under review as a conference paper at ICLR 2027

Information-Guided Representation Learning from GUI Pixels for Misinformation Detection

Abstract

Most misinformation detectors operate on text and metadata, whereas users encounter posts through rendered interfaces that also expose account cues and selected replies. Screenshots make these cues available but entangle decision-relevant evidence with visual appearance. We study GUI-derived representations using source-aware structured fields, a frozen screenshot encoder, and trainable content, identity, appearance, and exposure branches. The model keeps selected cross-type interactions explicit and applies a consistency objective to low-level transformations that preserve the post and its visible evidence. To separate event generalization from repeated event content, we construct a post-grouped split of 1,878 PHEME and Twitter15/16 posts into 183 adjudicated global event groups. On 360 posts from 37 held-out groups, a structured model using GUI-extracted and legacy-derived fields reaches 71.67% Macro-F1, versus 64.81% for extracted-text features and 54.01% for a frozen screenshot embedding alone. Yet ordinary fusion, which leads under a random-post split, falls behind structured fields on unseen events; neither typed interactions nor appearance-pair consistency closes this gap. This reversal shifts the focus from fusing more GUI information to identifying which interface-derived evidence generalizes across events.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI understanding, representation learning, multimodal learning, misinformation detection, social media analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_5HuyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates misinformation detection from the perspective of social‑media GUI screenshots. Instead of relying on raw text and metadata from platform APIs, it works with interface screenshots as actually perceived by end‑users. It decomposes information into four input categories: content, identity, appearance and exposure, and designs a structured representation scheme with selective invariance. It also constructs an event‑disjoint evaluation split and points out that random‑sample splitting overestimates model generalization performance. The topic is practically relevant, and the experiments are carefully designed, including manual extraction audits, multiple comparison baselines and ablation studies. However, the core proposed typed‑fusion model fails to outperform simple baselines in cross‑event generalization. The method yields limited gains; some conclusions remain descriptive, and several experimental and argumentative flaws exist.

Critical strengths

Most existing misinformation detection systems adopt backend‑stored platform text and user metadata. By contrast, this paper focuses on GUI screenshots that users actually view. Screenshots intermix content, account identifiers, comment exposure and visual styling. The work distinguishes evidential information from pure rendering‑style changes and introduces the concept of selective invariance. It argues that models should produce stable predictions for rendering‑only perturbations while allowing prediction shifts when underlying evidence changes. This problem framing itself is valuable.

Critical weaknesses
  1. The core Typed Fusion approach delivers no cross‑event generalization improvements; the practical benefit of the paper’s central method remains unclear. The key innovations are the four‑branch typed representation, explicit cross‑type interactions, and the selective‑consistency pair loss. Nevertheless, on the critical held‑out‑event test set:Structured GUI fields (directly using extracted structured fields rather than the learned typed fusion) achieve the best Macro‑F1 of 71.67%.Typed fusion without pair loss reaches only 64.98%. Adding pair loss further degrades performance to 63.41%, underperforming even ordinary four‑source fusion at 66.46%.

  2. Manual audits on 120 posts reveal imperfect GUI extraction: exact‑match accuracy for post text is merely 51.67%, and avatar category agreement stands at 70%. The model directly consumes these extracted outputs, but no oracle experiments are conducted. How would model performance change given human‑corrected, perfectly extracted features? To what extent does extraction noise alter model comparative rankings? Only error statistics are provided without quantifying noise impacts on end‑task performance, which weakens the reliability of conclusions.

  3. Figure 1 workflow diagram appears highly AI‑generated.

  4. Datasets are fairly outdated; no datasets from the past three years are adopted.

5.Inconsistent cross‑dataset performance raises concerns over the method’s generalizability.Slice‑based results show structured GUI fields substantially outperform text features on the PHEME dataset, yet underperform text features on Twitter15/16. Relative model performance reverses across datasets. This indicates that the effectiveness of this GUI‑driven method heavily depends on the event and label distribution inherent to each dataset. One cannot claim that GUI features universally outperform text. This limitation should be emphasized more prominently in the paper.

6.All experiments are built on simulated X/Twitter GUI rendering.

Questions for authors
  1. The core Typed Fusion approach delivers no cross‑event generalization improvements; the practical benefit of the paper’s central method remains unclear. The key innovations are the four‑branch typed representation, explicit cross‑type interactions, and the selective‑consistency pair loss. Nevertheless, on the critical held‑out‑event test set:Structured GUI fields (directly using extracted structured fields rather than the learned typed fusion) achieve the best Macro‑F1 of 71.67%.Typed fusion without pair loss reaches only 64.98%. Adding pair loss further degrades performance to 63.41%, underperforming even ordinary four‑source fusion at 66.46%.

  2. Manual audits on 120 posts reveal imperfect GUI extraction: exact‑match accuracy for post text is merely 51.67%, and avatar category agreement stands at 70%. The model directly consumes these extracted outputs, but no oracle experiments are conducted. How would model performance change given human‑corrected, perfectly extracted features? To what extent does extraction noise alter model comparative rankings? Only error statistics are provided without quantifying noise impacts on end‑task performance, which weakens the reliability of conclusions.

  3. Figure 1 workflow diagram appears highly AI‑generated.

  4. Datasets are fairly outdated; no datasets from the past three years are adopted.

5.Inconsistent cross‑dataset performance raises concerns over the method’s generalizability.Slice‑based results show structured GUI fields substantially outperform text features on the PHEME dataset, yet underperform text features on Twitter15/16. Relative model performance reverses across datasets. This indicates that the effectiveness of this GUI‑driven method heavily depends on the event and label distribution inherent to each dataset. One cannot claim that GUI features universally outperform text. This limitation should be emphasized more prominently in the paper.

6.All experiments are built on simulated X/Twitter GUI rendering. In real‑world scenarios, platform UI version updates, layout differences across clients, and dynamically loaded comments all alter visible GUI content. Cross‑platform and UI‑revision testing is absent. Furthermore, prediction is performed at the source‑post level. Real‑world screenshots come from heterogeneous sources and may not preserve all visible reply evidence, imposing strict constraints for practical deployment.

Reviewer_iuCuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies binary veracity prediction for social media posts rendered as X/Twitter-style screenshots. Each source post comes with up to three captured reply views: Relevant, Likes/Popular, and Recent. A frozen Qwen3-VL-4B pipeline extracts post text, account categories, and reply summaries. A predecessor rule system adds hand-set credibility and evidence scores (Appendix E).

The model groups its inputs into four types. Content is TF-IDF/SVD over extracted text and topic. Identity covers badge, avatar, and username categories plus a rule-derived credibility score. Appearance is a frozen SigLIP 2 embedding of the post screenshot. Exposure covers reply counts and rule-derived verification, consensus, social-ratio, and temporal scores. Typed fusion projects each type separately and adds three elementwise interactions. An optional pair loss penalizes prediction changes under low-level image transformations.

The evaluation uses 1,878 posts from PHEME and Twitter15/16, grouped into 183 adjudicated events. The held-out test set has 360 posts from 37 events. Structured fields reach 71.67% Macro-F1, extracted text reaches 64.81%, and the frozen screenshot embedding reaches 54.01%. Ordinary fusion (66.46%) outperforms typed fusion (64.98%) and pair-trained typed fusion (63.41%). Under a random-post split, the ranking reverses and fusion leads.

As I read it, the main contribution is an evaluation finding. The proposed typed representation and pair loss do not improve event-held-out performance.

The paper qualifies most claims carefully, but these qualifications are spread across Sections 3 to 5 and Appendices A to F. The reader must assemble them to see that the headline structured result depends on rule-derived scores, that key controls are missing, and that the test set was inspected. The abstract and contribution list do not state these limits.

Critical strengths
  1. An event-disjoint evaluation that changes the conclusion. The authors adjudicate 183 global event groups. They keep all views and variants of a post in one partition and compare against a random-post split. The ranking reversal is informative. The frozen screenshot model falls from 81.18% to 54.01% Macro-F1, and ordinary fusion falls from 84.23% to 66.46%. This is a useful warning for GUI-based and multimodal misinformation work, where random splits remain common.

  2. Candid reporting. The paper reports negative results for its own method. It also reports seed-level variation, event-resampled intervals, and a field-level extraction audit. Appendix E documents the hand-set rules behind every rule-derived input. This transparency is what makes the concerns below visible.

Critical weaknesses
  1. The headline structured result cannot be attributed to GUI evidence, and its evaluation is not independent. The structured model includes hand-set credibility, verification, consensus, social-ratio, and temporal scores from the predecessor pipeline. The authors state in Section 3 and Appendix D that a raw-field-only control is needed to attribute the gain. They do not report it. The main empirical claim therefore rests on a comparison the authors themselves describe as incomplete.

The independence of the test evaluation is also limited. The Reproducibility Statement and Appendix D state that the test set was inspected during development. Appendix F states that an earlier 2,000-configuration search selected its top settings by test accuracy. The paper does not specify which posts that search used. It also does not say whether any L2 to L4 constants came from it. If those constants were tuned on posts from the current test events, the structured scores may carry test information.

  1. Identity and exposure features may not reflect what users saw when the rumors spread. PHEME and Twitter15/16 posts date from 2014 to 2016. X introduced gold and gray checkmarks in December 2022 and reply sorting by Relevant, Recent, and Liked in August 2024. Yet the extractor assigns gold, gray, blue, or no badge (Appendix E, L1), and gold or gray badges receive the highest identity score of 1.00 (Table 7). Appendix E also states that badge color was not independently confirmed. The badge categories therefore likely reflect account status at capture time, not at posting time.

The same concern applies to replies. Appendix D states that no timestamped reply pool is available. The consensus summary records whether fact-checking comments are present (Appendix E, L4b). If replies were captured years after the events, they may include post-hoc debunks. This matters if the task targets detection before a claim is resolved. The paper does not state which setting it targets.

Event-disjoint splits also do not guarantee account-disjoint splits. In the prototype, removing the badge alone lowers accuracy by 12.78 points (Table 10). The paper does not report account overlap between training and test. These issues leave open whether the structured advantage reflects generalizable interface evidence or source and hindsight shortcuts.

  1. The proposed method shows no benefit, and the framing does not match the findings. Typed fusion, typed fusion without interactions, and pair-trained typed fusion all trail ordinary fusion on the primary metric. The pair loss shows no reliable flip-rate gain, and its interval crosses zero. Ordinary fusion already flips only 0.63% of predictions, which leaves little room to show a gain. The evidence-changing side of selective invariance is never tested. The title emphasizes representation learning from GUI pixels. However, the pixel encoder is frozen, and the screenshot branch is the weakest input at 54.01% Macro-F1. The contribution list also presents the typed representation as a main contribution, although it does not improve the primary metric.

  2. The evidence base is thin, and the comparisons are not controlled. The test set has 37 events, and one event contains 163 of the 360 posts. The interval for structured minus text is [-11.50, +13.04], so the gain over text is not established. Text outperforms structured fields on Twitter15/16 (72.97% vs. 69.81%). The structured model has 3,897 parameters, while the fusion models have 103K to 137K parameters and train on 1,297 posts. The paper notes that these models are not parameter-matched. Capacity and regularization therefore remain alternative explanations for the ranking. The text baseline uses TF-IDF over VLM-extracted text with a 51.67% exact-match rate. The paper does not quantify how extraction errors affect the text baseline.

Questions for authors

Q1. Screenshot provenance. Were the posts and reply views captured from the live site or rendered from dataset records? On what dates? How were gold and gray badges assigned to accounts from 2014 to 2016? How many posts were lost to deletion, suspension, or protected accounts?

Q2. Rule constants. Which posts did the 2,000-configuration search in Appendix F use? Were any L2 to L4 weights, thresholds, caps, or multipliers chosen or revised after viewing results on posts from the 37 test events?

Q3. Target setting and reply timing. Does the task target detection before a claim is resolved, or retrospective verification? What fraction of captured replies were posted after the claim was resolved? Does the structured advantage hold without the consensus and temporal scores?

Q4. Account overlap. How many test-set source accounts also appear in training? What are the results on test posts from accounts unseen in training?

Q5. Labels. How were the non-rumor and unverified classes of PHEME and Twitter15/16 mapped to the binary label? What is the source of the external label file? Why do 296 posts disagree with the predecessor's labels? Were the predecessor's rules developed with those labels?

Suggestions for strengthening the paper (not conditions for acceptance)

S1. Report the raw-field-only control and a text control that uses the original dataset text. These would separate extraction noise, rule scores, and raw GUI cues.

S2. Use repeated grouped cross-validation over events with event-clustered intervals. Since the current test set has been inspected, a fresh held-out set would give a cleaner confirmation.

S3. Add a stronger text baseline on the same split, such as BERTweet (Nguyen et al., EMNLP 2020 System Demonstrations). Match model capacity or tune regularization for the fusion models.

S4. Consider reframing the paper as an evaluation study and revising the title and contribution list to match the findings.

Reviewer_EeDVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper reframes social-media misinformation detection as learning from platform-rendered observations (screenshots) rather than machine-readable API records. Users see account badges, avatars, engagement counts and selected replies, so a screenshot is not merely a picture of a record but the interface through which a post is consumed. The authors operationalize this with four addressable input types — content (TF–IDF/SVD over post text), identity (one-hot account attributes plus a predecessor credibility score), appearance (a frozen SigLIP-2 embedding of the source-post screenshot), and exposure (comment counts and rule-derived view summaries) — combined through type-specific projections, three hand-picked element-wise interactions, and a two-layer MLP, optionally trained with a consistency loss over appearance-preserving rendering variants ("selective invariance").

Critical strengths

The paper poses an interesting question — what misinformation cues are exposed by a rendered interface rather than by an API record — and its honest reporting of negative results is commendable.

Critical weaknesses
  1. Limited algorithmic novelty — mostly a fusion of existing feature methods. The proposed "typed representation" is, in substance, a concatenation of four already-standard feature families: TF–IDF+SVD over post text (content), one-hot/standardized account attributes (identity), a frozen SigLIP-2 embedding of the screenshot (appearance), and rule-derived comment counts and scores (exposure). The "learning" is limited to per-branch linear/GELU projections, three hand-picked element-wise products, and a two-layer MLP, plus an MSE pair-consistency loss that is essentially a standard consistency/robustness regularizer (cf. AugMix-style training the authors themselves cite).

  2. Weak and incomplete experimental design. The headline result actually undercuts the proposal: ordinary four-source fusion (66.46%) beats typed fusion (64.98%), and the pair objective hurts performance (63.41%, with large seed variance: 67.90/65.44/56.89). The most informative row (71.67%) is a structured baseline that includes inherited rule scores, yet the decisive raw-field-only and human-corrected-field controls are explicitly deferred as "remaining" (Appendix D). Without them, the gain cannot be attributed to the architecture versus legacy engineered features. Likewise, the appearance-consistency test covers only JPEG/scale/contrast/border perturbations and its bootstrap interval crosses zero — selective invariance is asserted, not demonstrated. The event-clustered bootstrap intervals are described as post-hoc and "not confirmatory," which is the right caveat but leaves little rigorous statistical support.

  3. Small data scale. The whole study rests on 1,878 posts (1,290 PHEME + 588 Twitter15/16), and the held-out event test is only 360 posts from 37 groups, with a single group holding 163 posts. Three seeds and a ± seed-standard-deviation are reported rather than event-clustered confidence intervals. Such a small, skewed sample makes the reported 5–7 point differences fragile and hard to distinguish from sampling noise, especially given the observed seed instability.

  4. Missing comparisons to relevant methods. There is no comparison against established misinformation/rumor detectors on a common protocol — the authors cite dEFEND, GCAN, GenFEND, EANN/MVAE, and VizFake but never run them (or even text/multimodal SOTA baselines) on their split. The only "baselines" are ablations of their own pipeline plus a few poorly-prompted VLM/LLM rows (Qwen2.5-7B at 26.5% accuracy) that the paper concedes are "not controlled tests." This makes it impossible to situate the contribution relative to prior work.

Questions for authors

To strengthen the paper: add a raw-field-only / oracle-field ablation, expand the dataset and report event-clustered confidence intervals, include matched comparisons to prior fusion/multimodal detectors, and either demonstrate a real advantage of typed fusion or reposition the contribution as an honest negative/protocol study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.