acceptodds
Under review as a conference paper at ICLR 2027

What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media

Abstract

Public social media posts can reveal private information through clues in text and images. Details that seem harmless in a single post can jointly expose a user's home address, workplace location, or daily routine when combined across posts. However, current research lacks a shared benchmark for user-level multimodal privacy leakage and an evaluation metric that captures exposure severity beyond binary accuracy, hindering consistent comparisons across methods. To support cross-post evaluation, we introduce SopriBench, a synthetic benchmark that groups posts by user and provides ground-truth private attributes with post-level supporting evidence. Guided by leakage patterns identified on public social media, it contains 50 user profiles, 500 posts, and 1,569 images. The accompanying Privacy Exposure Score (PES) scores correctly inferred information by its specificity and sensitivity in the user's context. To connect clues across posts, we develop Argus, a training-free agentic framework that keeps track of possible attribute values, verifies supporting evidence, and assembles user-level privacy profiles. On SopriBench, Argus achieves a 25% relative PES gain over the strongest baseline, SingleAgent. In a study where 11 users verify inferences about themselves, it achieves 75.5% precision, compared with 51.4% for SingleAgent. An ablation shows that removing verification increases binary accuracy but lowers the average specificity score.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Social Media Privacy, User-level Privacy Inference, Multimodal Benchmarking, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_2oCpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates cumulative privacy leakage from public social media posts, where seemingly harmless textual and visual clues can be combined to infer private user attributes. The authors make three main contributions: (1) SopriBench, a synthetic benchmark containing 50 user profiles, 500 posts, and 1,569 images, with annotations for private attributes, supporting evidence, and leakage characteristics; (2) Privacy Exposure Score (PES), a metric that evaluates inferred attributes based on their granularity and contextual sensitivity; and (3) Argus, a training-free agentic framework that maintains hypotheses, aggregates cross-post evidence through a graph structure, and verifies inferences using external tools.

Experiments show that Argus achieves a 25% relative PES improvement over the strongest baseline, SingleAgent, with particularly notable gains on cross-post leakage. The authors further provide ablation studies, human annotation audits, and a small-scale real-user validation involving 11 participants to assess the framework's effectiveness and evaluation reliability.

Critical strengths

1. In-depth Privacy Scenario. The paper provides a thorough investigation of privacy leakage in public social media, moving beyond isolated disclosures to examine how seemingly harmless clues accumulate across posts, modalities, and contextual information. I particularly appreciate the authors' effort to characterize this specific scenario in depth, including different leakage mechanisms, inference difficulties, and contextual sensitivity. This focused treatment makes the problem setting meaningful and relevant to real-world privacy risks.

2. Substantial Research Effort. The amount of work invested in this study is commendable. The authors analyze a substantial real-user reference corpus, construct a multimodal synthetic benchmark with detailed annotations, and develop a multi-stage agentic framework. They also conduct baseline comparisons, ablation studies, human annotation audits, and real-user validation. These efforts strengthen the credibility of the study.

3. Well-motivated Agentic Framework. Argus explicitly maintains unresolved hypotheses, collects additional evidence through external tools, and verifies inferences before incorporating them into user profiles. This design is well aligned with the cumulative nature of privacy leakage. The improvements on cross-post inference tasks provide empirical support for the value of structured evidence management and iterative verification.

Critical weaknesses
  1. [Major] Incomplete Case Study. Although the paper provides extensive methodological details and a qualitative example in Figure 3, I would appreciate a complete, concrete case study illustrating the entire pipeline. Showing the original multimodal posts, ground-truth attributes, extracted evidence, hypothesis updates, tool outputs, verification decisions, and final privacy profile would make the framework easier to understand and assess. Given the complexity of the system, I consider such an end-to-end example important.

  2. [Minor] Complexity and LLM Dependence. The framework involves numerous components, models, and LLM-based decisions throughout data construction, inference, verification, and evaluation. While this is not necessarily a methodological flaw, I am concerned that the overall design may be unnecessarily complicated and introduce multiple sources of model-dependent bias. Despite the human evaluation audits, the robustness of the conclusions to different model choices, prompts, and evaluation procedures remains insufficiently explored.

  3. [Minor] Synthetic Benchmark Validity. SopriBench is carefully constructed, but its privacy leakage patterns are explicitly designed through predefined user profiles and leakage plans. This raises concerns about whether recovering these planned attributes adequately reflects the uncertainty and ambiguity of real-world privacy inference. Although the real-user validation is encouraging, its limited scale makes it difficult to fully establish the benchmark's generalizability.

  4. [Minor] Insufficiently Controlled Comparisons. Argus substantially outperforms SingleAgent, but it also uses considerably more model calls, tool calls, and execution time. Therefore, it is difficult to isolate how much of the improvement comes from the proposed reasoning framework rather than additional computational effort. Moreover, the CPeg ablation compares graph-based memory against conversation history rather than an equally informative structured-memory baseline. More carefully controlled comparisons would strengthen the claims about the framework's effectiveness and novelty.

  5. [Minor] Subjectivity in Agent Design. Several core components of Argus appear to rely on heuristic or LLM-driven decisions, including hypothesis prioritization, tool routing, evidence verification, and hypothesis acceptance. Although the authors describe the general principles and fix the routing policy before evaluation, the operational criteria behind these decisions remain insufficiently specified. In particular, it is unclear how confidence scores are determined, what constitutes sufficient evidence, and how subjective decisions affect the final results. I would appreciate clearer design justifications and analyses demonstrating the robustness of these choices.

Questions for authors

My main questions and concerns are detailed in the Weaknesses section.

I appreciate the authors' in-depth investigation of social media privacy leakage and the substantial effort invested in this work. However, my primary concern is the lack of a complete end-to-end case study, which makes it difficult to fully understand and assess the proposed framework. If the authors can provide one or more concrete examples illustrating the complete reasoning and verification process, I would be happy to reconsider my assessment and raise my score.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 2oCp (1/3)

Dear Reviewer 2oCp,

Thank you for your detailed comments on the major and minor points and for recognizing our investigation of cumulative leakage and the motivation for evidence management.

We have uploaded the latest revised PDF. References use the revised numbering unless we specify the original submission.

W1. End-to-end case study of Argus

Figure 3 of the original submission illustrates residence inference. To show the full process, we added a complete end-to-end case study of Argus and SingleAgent in revised Section 5.2 and Appendix D.1. It includes post excerpts, ground-truth attributes, extracted evidence, hypothesis updates, tool outputs, verification decisions, and final privacy profiles. Figure 3 compares the two methods at successive steps of the home and workplace investigations. It uses redacted source images, including a magnified crop of the vehicle app's location field. The venue map, notice, outing photo, and folder crest show the recorded cues.

StageObservation and decision
ResidenceP2 compares three wedding venues near home. Argus stores their locations and the near-home cue in CPeg and links the P9 invitation to the selected venue. The cluster narrows the search, while SingleAgent carries forward the city. The delivery map leaves two sections of the same apartment complex plausible. SingleAgent keeps Section 1 after reading the notice. Argus matches the building and unit prefix on the earlier waybill to the notice, resolving Unit R in Building K, Section 2.
WorkplaceBoth methods see the parking area in the vehicle app's top bar. Argus records it as a vehicle location. SingleAgent combines parking with a teaching qualification to guess nearby School T. Both see the spring outing post. Argus matches the Grade 3 Class 2 sign, folder crest, and area with school records to identify School S and the class, while SingleAgent keeps School T.

The case shows how Argus keeps address candidates open, keeps parking and qualification clues separate from employment, and checks later evidence before accepting an early guess.

Appendix D.1 gives the annotated values and lists all nine post excerpts in Table 27. It records the tool observations, the first incorrect choices, how each method treats the later evidence, and the final profiles.

W2. System complexity and evaluator choice

Figure 2 of the original submission shows the initial skim, targeted investigation, and evidence verification. The same submission reports 0.7 deep visual inspections out of 13.3 routed actions per user (Appendix B.7, Table 17) and describes human checks for cue plausibility and unintended identifiers (Appendix A.3). In its sensitivity audit, three annotators independently rescored 78 of 781 leaked attributes, with ordinal agreement of 0.852 (Appendix A.2.2, Table 7). Its evaluator audit shows that the six-method ranking also holds with adjudicated human labels (Appendix C).

To check whether Argus's lead depends on the evaluator, we added a comparison with two alternative models in revised Section 5.3 (Table 4). We rescored the 546 decisions on 78 leaked attributes from the original human audit (Appendix C) with DeepSeek-V4.1-Flash and GLM-5.3-Flash. Both received the same predictions, annotations, and scoring rules as GPT-5.5, with method names withheld. We compared their slot matches and granularity scores with the adjudicated human labels and computed both methods' PES on the same subset.

EvaluatorSlot agreement (%)Slot Cohen's κGranularity agreement (%)SingleAgent PESArgus PES
GPT-5.594.00.87991.20.450.56
DeepSeek-V4.1-Flash92.70.85389.20.440.55
GLM-5.3-Flash91.80.83589.00.450.54

The Argus–SingleAgent PES gaps under the two evaluators are 0.11 and 0.09, respectively. Appendix C of the original submission describes the main scoring procedure.

The original submission reports a shared-model comparison of Argus and SingleAgent (Section 5.2 and Appendix B.7, Table 16). With Qwen3.6-35B-A3B, Argus scores 0.50 PES compared with 0.39 for SingleAgent. To control for model choice across methods, we extended this comparison to all six methods in revised Appendix E.2.

We ran a separate experiment to see whether Argus's lead depends on the wording of the prompts (Appendix E.2). We changed one prompt setting at a time, with the same inputs and resource limits. Argus scores 0.53–0.56 PES, compared with 0.43–0.45 for SingleAgent, with paired gaps of +0.10 to +0.12. Argus leads SingleAgent in each prompt setting.

AuthorsOpenReview ↗

Response to Reviewer 2oCp (2/3)

W3. Real-world validity of the synthetic benchmark

The original submission describes how we use patterns from 318 reference accounts to guide benchmark construction and how the leakage plans specify target attributes and clues (Sections 3.1–3.2). It also reports a 104-response study of profile authenticity and image discrimination (Appendix A.3).

To check what the synthetic posts reveal, we added a comparison of clue patterns and an audit of support for the annotated attributes in the revision (Section 3.2 and Appendix A.3.1). We used the same clue carrier and cross-post relation categories for the reference and synthetic accounts. In the audit, we checked whether the synthetic cues were visible, consistent across posts, and sufficient to support the annotated value.

CheckResult
Synthetic age, location, occupation, and educationEvery synthetic ground-truth profile contains all four fields.
Reference versus synthetic cue carriers and cross-post relationsThe overlap is 0.90 for clue carriers and 0.87 for cross-post relations.
Synthetic cue visibility, consistency, and supportVisibility is 94.0% (470/500 posts), consistency is 91.6% (458/500 posts), and full cross-post support is 95.7% (379/396 D4 attributes).

To check the account mix, we also compared age and work or study status across the reference and synthetic accounts (Appendix A.3.1, Table 14). We used ten posts per account and recorded only explicitly stated information. Missing or ambiguous values are marked Unknown. Employed accounts make up 52.0% of the synthetic set, compared with 42.2% of real Rednote and 45.2% of real Instagram. Unknown work or study status accounts for 4.0%, compared with 15.5% and 12.7%. All age and work/study categories are represented.

We then compared how often clues about each type of private information appear in the two sets of accounts. We use real accounts for aggregate checks and synthetic data for ground-truth evaluation. The table reports the share of accounts with at least one clue in each privacy category.

Privacy categoryReal reference share (%)Synthetic share (%)
Identity documents or ID-related information21.7%28.0%
Home address56.6%70.0%
Occupation and workplace65.7%82.0%
Education and study history48.7%58.0%
Income, finance, and assets42.8%46.0%
Travel and daily routines69.5%76.0%

We also ran a questionnaire to study unintended disclosure in everyday sharing. We examined which privacy clues participants might overlook when deciding what to share and what they could infer from the same synthetic posts. We collected 19 responses (Appendix A.3.1). We asked participants which posts they would share on a public account and which details they would change, crop, or omit, with their reasons. Participants rated typicality on a 1–5 scale, flagged details that seemed inconsistent or deliberately placed, and gave the most specific occupation they could infer with supporting post IDs.

Questionnaire measureResult
Post as-is26.3%
Edit or omit content68.4%
Mean typicality score (1–5)4.2
Inconsistency-flag rate10.5%
Correct occupation-inference rate84.2%
Correct supporting-post identification rate78.9%
Deliberately-placed-detail flag rate15.8%
Other / do not post or unsure5.3%

To check whether the gains extend across user groups, we added a performance breakdown by work or study status (Appendix D, Table 25).

GroupShare (%)AttributesSingleAgent PESArgus PESSingleAgent cross-post PESArgus cross-post PES
Student20.0%1560.450.560.400.53
Employed52.0%4060.430.540.380.50
Self-employed18.0%1400.450.560.410.53
Retired6.0%47————
Unknown4.0%32————
Total100.0%7810.440.550.390.51

Argus leads in all three groups with at least eight accounts. Employed is the largest group and has the lowest SingleAgent PES of the three groups (0.43). Retired and Unknown are not interpreted separately because each has fewer than eight accounts.

In the 11-user validation reported in the original submission (Section 5.2 and Appendix C, Table 19), Argus reaches 75.5% confirmed precision and 11.2 confirmed attributes per user, compared with 51.4% and 9.9 for SingleAgent. This provides a direct check on those accounts alongside the controlled synthetic results. Thank you again for specifying the evidence needed to assess the framework.

AuthorsOpenReview ↗

Response to Reviewer 2oCp (3/3)

W4. Budget matching and structured memory baselines

Resource budgets. The original submission reports full-run costs (Appendix B.7, Table 15). To test whether Argus's gains persist at comparable resource use, we added comparisons at separate token, tool-call, and time limits in revised Section 5.4 (Table 5). For each user, SingleAgent decides when to stop, and its completed run sets separate limits for tokens, tool calls, and time. The paired runs use the same model, posts, and tools. For each limit, we build the Argus profile from evidence accepted by the last investigation round completed within the limit. We measure time from the start of each run to SingleAgent's final profile or the end of the selected Argus round. Token counts include model input and output. We score each profile as in the main comparison, reuse saved judgments for unchanged scoring inputs, and assign zero to omitted attributes.

Argus scores higher at all three limits. At the time limit, it averages 4.7 minutes per user compared with 4.8 for SingleAgent, with gains of 0.02 overall PES and 0.03 cross-post PES. Tokens, tool calls, and time in the table are per-user averages.

MethodTokensTool callsTime (min)PESCross-post PES
SingleAgent86.5K5.14.80.440.39
Argus (tokens)85.7K6.36.50.480.45
Argus (tool calls)82.6K5.16.20.470.44
Argus (time)81.9K4.84.70.460.42

Revised Appendix B.7 describes the replay and scoring protocol.

The original submission reports the ablation that replaces CPeg with conversation history (Section 5.4, Table 3). Revised Section 5.3 (Table 3) reports a drop in cross-post PES from 0.51 to 0.38.

To test the effect of evidence organization, we compared CPeg with a structured ledger in an additional experiment (Section 5.3 and Appendix E.2). Both received the same saved observations, provenance, hypotheses, and support links. We kept the model, verification criteria, and context allowance fixed. The ledger yields 0.50 PES, 0.42 cross-post PES, 74.5% precision, and 104.0K tokens per user. The corresponding CPeg results are 0.55, 0.51, 84.8%, and 98.6K.

W5. Hypothesis prioritization, tool routing, and evidence requirements

Hypothesis prioritization, routing, and admission are governed by explicit criteria. At each round, Argus first investigates the highest-priority unresolved hypothesis. The original submission describes the hypothesis store, routing policy, and evidence verification in Appendices B.2–B.4. The router considers missing evidence, the likely value of a route, the remaining budget, and prior attempts. It uses fixed rules for common routes and an LLM fallback when no rule matches. Admission is a separate decision about whether the source is reliable and the chain supports the claimed value at the stated specificity. For example, identifying the compound in a photo establishes its location. A claim that the user lives there requires evidence that links the user to the compound.

In the original Verifier ablation (Section 5.4, Table 3), removing the Verifier raises coverage from 0.71 to 0.74, lowers granularity from 0.60 to 0.57, and lowers PES from 0.55 to 0.50.

To see how the results depend on these design choices, we added an experiment that changes one investigation rule at a time (Appendix E.2, Table 29). We tested alternatives for hypothesis ordering, routing fallback, and evidence admission, with inputs and resource limits fixed.

VariantPESPrecision (%)False positives/userTokens/user
Default0.5584.81.16117.9K
Fixed hypothesis order0.5079.52.20116.0K
Fixed tool selection rules0.4776.32.84109.1K
Independent evidence required0.5387.81.20125.7K

Thanks again for your thoughtful reviews and your time and effort to our community!

Best,
Authors

Reviewer_XnyaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The work studied user-level privacy leakage in multimodal setting from synthesized profiles, grounded on real social media users. The authors propose two major contributions: (1) SopriBench, a synthetic benchmark with 50 users, 500 posts, and 1,569 images, using a Privacy Exposure Score (PES) as a metric, which accounts for both inference context-based sensitivity and granularity; (2) Argus, a training-free agentic framework that performs cross-post inference using explicit hypotheses, evidence verification, and a Cross-Post Evidence Graph (CPeg). The proposed Argus method achieves strongest PES score against the existing baselines, like SingleAgent and provide various ablations on impact of various inference framework parts. The graph-based CPeg feature is proven to substantially benefit the cross-post inference performance. The paper addresses an important problem and presents a coherent system design, supported by IRB-approved human study. Lastly, the SopriBench benchmark can be reproduced from covered findings, however, there are concerns regarding limited access to Argus framework codebase to validate the presented positive gains.

Critical strengths

S1. The motivation of the work is clear and justified, providing an extensive picture on multimodal private attribute inference evaluation. It also focuses on a very important aspect of this task - PII inference should be done across posts, since the clues are usually scattered across different social medias in various forms, providing much more realistic simulation of deanonymization process in the wild.

S2. The work introduces Argus, a training-free agentic framework to conduct cross-post personal inference and the CPeg graph idea is particularly useful with numerous novel features. The method shows strong performance compared to other inference frameworks, including solid multi-step process with hypothesis verification, showing the benefit of the cumulative cross-post inference pipeline. The inference efficiency was verified by real users by a well-thought IRB-approved human study in a privacy-preserving manner, which is an extremely strong proof of method and benchmark quality.

S3. The proposed SopriBench is well-designed, presenting wide range of social interaction and presence types; the benchmark enables the reproducible and controlled experiments without releasing the underlying real-user content. The benchmark generation pipeline clearly shows substantial effort investment and level of design rigor.

Critical weaknesses

W1. Argus gains may come just from the fact that very strong models are used. There is an ablation study comparing with SingleAgent using the same model, but there were much more costs involved for Argus, which can be correlated with performance increase. The work would benefit from more thorough cost-performance analysis of all methods in a standardized setup. Moreover, models can be stereotypically biased and prone to guessing based on stereotypical reasoning, so the work could also be improved by a qualitative analysis of inference behavior for specific models.

W2. The size of SopriBench set is not very big user-wise, but has high number of posts. However, lacking users results in lower diversity and potentially lower average difficulty of profile inference cases. Furthermore, included study shows that the synthetic image quality is realistic, but this does not mean that images leak the same features in similar fashion across many posts as original ones.

W3. The study lacks the distribution information on user set and how different are synthetic users from real ones. Also, the Argus codebase is planned to be partially public at limited access, which blocks the reproducibility of method, thus making it hard to verify how good the method actually is. There are be enough materials to run and reproduce the SopriBench benchmark, but not Argus as a novel strong method proposal

Questions for authors

Q1. Argus shows strong gains and conceptually the method makes a lot of sense. However, can the authors please provide more insight on the pure impact of method compared to all methods (and whether using older or open-weight models shows the benefit compared to all listed methods) and whether hypothesis-based approach does not gain purely from higher budget? It is also not very clear how the other methods were ran from technical POV in the main comparison (for reproducibility purposes). Apppendix B Table 15 shows that Argus is using 6x more calls than SingleAgent, so ruling out the possibility of having bigger budget giving higher scores would help the method presentation greatly.

Q2. I might not have understood it well from the writing, but can some private attributes be hypothesized across multiple posts and do hypotheses come on single post-level? (e.g. mentioning "buying treats" or "grooming" which do not necessarily mean having a pet, but appearing across multiple posts can be counted as a valid hypothesis).

Q3. 28 free-mode private attributes is quite a lot (more than related work), which may pose the risk of lacking standardization for reproducible and robust benchmarking - did you notice any discrepancies w.r.t inference answer verification? For example, education or income level types can be very diverse and vary depending on country of origin.

Major (addressing these would help me raise my score, higher priority): W3; W1+Q1: appendix C - why use GPT-5.5 as LLM judge and not multi-judge setup to avoid potential bias?

Minor (does not affect my score, only personal recommendations): no limitations section

6 comments
AuthorsOpenReview ↗

Response to Reviewer Xnya (1/3)

Dear Reviewer Xnya,

Thank you for your detailed review and for recognizing the motivation, benchmark construction, and cross-post evidence design. We appreciate your questions about resource use, evaluator choice, and reproducibility.

We have uploaded the latest revised PDF. References use the revised numbering unless we specify the original submission.

Major

W1 / Q1. Effects of model choice and resource budgets

Model choice. The original submission reports that Argus leads SingleAgent by 0.11 PES when both use Qwen3.6-35B-A3B (Section 5.2 and Appendix B.7, Table 16). To control for model choice across the full comparison, we extended the experiment to all six methods in revised Appendix E.2. Each uses the same open-weight model and keeps its post inputs and tool access from the main comparison. The PES scores are 0.18 for TextLLM, 0.31 for PostVLM, 0.26 for SelfDisc, 0.34 for Holmes, 0.39 for SingleAgent, and 0.50 for Argus.

Baselines and reproducibility. Appendix B.8 of the original submission already describes each baseline's inputs, tools, and how it combines predictions (Table 18).

Resource budgets. The original submission reports full-run costs (Appendix B.7, Table 15). To test whether Argus's gains persist at comparable resource use, we added comparisons at separate token, tool-call, and time limits in revised Section 5.4 (Table 5). For each user, SingleAgent decides when to stop, and its completed run sets separate limits for tokens, tool calls, and time. The paired runs use the same model, posts, and tools. For each limit, we build the Argus profile from evidence accepted by the last investigation round completed within the limit. We measure time from the start of each run to SingleAgent's final profile or the end of the selected Argus round. Token counts include model input and output. We score each profile as in the main comparison, reuse saved judgments for unchanged scoring inputs, and assign zero to omitted attributes.

Argus scores higher at all three limits. At the time limit, it averages 4.7 minutes per user compared with 4.8 for SingleAgent, with gains of 0.02 overall PES and 0.03 cross-post PES. Tokens, tool calls, and time in the table are per-user averages.

MethodTokensTool callsTime (min)PESCross-post PES
SingleAgent86.5K5.14.80.440.39
Argus (tokens)85.7K6.36.50.480.45
Argus (tool calls)82.6K5.16.20.470.44
Argus (time)81.9K4.84.70.460.42

Revised Appendix B.7 describes the replay and scoring protocol.

W3. Argus code and method reproducibility

We have submitted a ZIP containing the Argus reference implementation and evaluation scripts as supplementary material. The full SopriBench synthetic dataset is too large to attach on OpenReview. After publication, we will make the synthetic dataset and evaluation scripts available through controlled access. Qualified researchers may apply for access to the dataset and the Argus code repository for approved academic evaluation. The Ethical Considerations section describes the Argus access procedure.

Q4. Evaluator diversity

Thank you for raising this concern. Appendix C of the original submission specifies the same GPT-5.5 rubric for attribute matching and granularity scoring across methods. Contextual sensitivity comes from fixed benchmark annotations.

To check whether Argus's lead depends on the evaluator, we added a comparison with two alternative models in revised Section 5.3 (Table 4). We rescored the 546 decisions on 78 leaked attributes from the original human audit (Appendix C) with DeepSeek-V4.1-Flash and GLM-5.3-Flash. Both received the same predictions, annotations, and scoring rules as GPT-5.5, with method names withheld. We compared their slot matches and granularity scores with the adjudicated human labels and computed both methods' PES on the same subset.

EvaluatorSlot agreement (%)Slot Cohen's κGranularity agreement (%)SingleAgent PESArgus PES
GPT-5.594.00.87991.20.450.56
DeepSeek-V4.1-Flash92.70.85389.20.440.55
GLM-5.3-Flash91.80.83589.00.450.54

The Argus–SingleAgent PES gaps under the two evaluators are 0.11 and 0.09, respectively.

AuthorsOpenReview ↗

Response to Reviewer Xnya (2/3)

Minor

W2. User diversity, inference difficulty, and realism of synthetic leakage

The original submission describes the 318 public reference accounts used to guide benchmark construction, including 161 from Rednote and 157 from Instagram (Section 3.1). It describes 50 synthetic users with ten Rednote-style posts each (Section 3.2) and reports 396 of 781 leaked attributes in the cross-post difficulty group (Appendix D). Its 104-response study reports profile authenticity ratings of 3.48 to 3.94 out of 5 and binary photo discrimination accuracy of 61.4% (Appendix A.3).

To check what the synthetic posts reveal, we added a comparison of clue patterns and an audit of support for the annotated attributes in the revision (Section 3.2). We used the same clue carrier and cross-post relation categories for the reference and synthetic accounts. In the audit, we checked whether the synthetic cues were visible, consistent across posts, and sufficient to support the annotated value.

CheckResult
Synthetic age, location, occupation, and educationEvery synthetic ground-truth profile contains all four fields.
Reference versus synthetic cue carriers and cross-post relationsThe overlap is 0.90 for clue carriers and 0.87 for cross-post relations.
Synthetic cue visibility, consistency, and supportVisibility is 94.0% (470/500 posts), consistency is 91.6% (458/500 posts), and full cross-post support is 95.7% (379/396 D4 attributes).

To check the account mix, we also compared age and work or study status across the reference and synthetic accounts (Appendix A.3.1, Table 14). We used ten posts per account and recorded only explicitly stated information. Missing or ambiguous values are marked Unknown and are not inferred from appearance or writing style.

CategoryRednote (%)Instagram (%)Synthetic share (%)
18–2524.8%22.9%20.0%
26–3029.8%29.9%30.0%
31–4025.5%26.8%30.0%
41–5510.6%12.1%14.0%
56+2.5%3.2%4.0%
Unknown age6.8%5.1%2.0%
Student20.5%16.6%20.0%
Employed42.2%45.2%52.0%
Self-employed16.8%18.5%18.0%
Retired5.0%7.0%6.0%
Unknown work or study status15.5%12.7%4.0%

The synthetic set includes every age and work/study category. Employed accounts are more common than in the real Rednote (52.0% versus 42.2%) and Instagram (45.2%) sets. Unknown work or study status is less common in the synthetic set (4.0% versus 15.5% and 12.7%).

We then compared how often clues about each type of private information appear in the two sets of accounts. We use real accounts for aggregate checks and synthetic data for ground-truth evaluation. The table reports the share of accounts with at least one clue in each privacy category.

Privacy categoryReal reference share (%)Synthetic share (%)
Identity documents or ID-related information21.7%28.0%
Home address56.6%70.0%
Occupation and workplace65.7%82.0%
Education and study history48.7%58.0%
Income, finance, and assets42.8%46.0%
Travel and daily routines69.5%76.0%

To see how readers interpret the synthetic posts, we ran a questionnaire study with 19 responses (Appendix A.3.1). We asked participants which posts they would share on a public account and which details they would change, crop, or omit, with their reasons. Participants rated typicality on a 1–5 scale, flagged details that seemed inconsistent or deliberately placed, and gave the most specific occupation they could infer with supporting post IDs.

Questionnaire measureResult
Post as-is26.3%
Edit or omit content68.4%
Mean typicality score (1–5)4.2
Inconsistency-flag rate10.5%
Correct occupation-inference rate84.2%
Correct supporting-post identification rate78.9%
Deliberately-placed-detail flag rate15.8%
Other / do not post or unsure5.3%

To check whether the gains extend across user groups, we added a performance breakdown by work or study status (Appendix D, Table 25).

GroupShare (%)AttributesSingleAgent PESArgus PESSingleAgent cross-post PESArgus cross-post PES
Student20.0%1560.450.560.400.53
Employed52.0%4060.430.540.380.50
Self-employed18.0%1400.450.560.410.53
Retired6.0%47————
Unknown4.0%32————
Total100.0%7810.440.550.390.51

Argus leads in all three groups with at least eight accounts. The largest group, Employed, has the lowest SingleAgent PES of the three groups (0.43). Retired and Unknown are not interpreted separately because each has fewer than eight accounts.

AuthorsOpenReview ↗

Response to Reviewer Xnya (3/3)

Q2. Hypothesis formation from clues across posts

Yes. Argus forms hypotheses across posts. It skims the full post collection before forming hypotheses from the evidence graph and revisits unresolved candidates as accepted evidence is added. Separate posts about buying treats and grooming can jointly motivate a pet hypothesis. Evidence linking the pet to the user is needed before it enters the final profile. Sections 4.2 and 4.4 of the original submission describe how Argus maintains hypotheses and decides which attributes enter the final profile.

Q3. Consistency of scoring free-form attribute predictions

The original submission describes the common definitions and scoring rules for the 28 attributes (Section 3.3 and Appendix C). The evaluator first matches each free-form prediction to an attribute and then compares the predicted value with the annotation.

For education, the evaluator gives credit at the deepest correct level in the annotated hierarchy. Equivalent degree names match at the same level. A prediction that gives just the degree receives degree-level credit when the annotation also specifies the institution and field. For income, the evaluator checks for an exact match in meaning, including the currency and time period when specified. A prediction of 3500 RMB/year receives zero when the annotation specifies 3500 RMB/month.

To quantify scoring disagreements for education and income, we added a breakdown of the original human audit in revised Appendix C. We compared GPT-5.5's judgments on these attributes with the adjudicated human labels. For education, 3 of 42 decisions (7.1%) disagreed on the attribute match and 4 of 42 (9.5%) on the value score. For income, the corresponding counts were 1 of 35 (2.9%) and 5 of 35 (14.3%).

Q5. Discussion of the study's limitations

We discuss both limitations explicitly in revised Section 6 (Conclusion and Limitation). The fixed window keeps the amount of account history comparable across methods. Longer histories can add supporting or conflicting clues. A larger real-user study would require substantial effort to protect participant privacy and verify inferences.


Thanks again for your thoughtful reviews!

Best,
Authors

Reviewer_S2rDOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SopriBench, a synthetic benchmark for studying user-level privacy leakage on social media, focusing on sensitive information distributed across multiple posts and modalities rather than disclosed within individual posts. Informed by patterns observed in real public social media posts, the benchmark comprises 50 synthetic user profiles, 500 posts, and 1,569 images. Each profile contains sensitive attributes deliberately exposed through distributed cues across the generated posts.

The paper also proposes PES, an evaluation metric that accounts for inference correctness, specificity, and information sensitivity, distinguishing coarse-grained from more specific predictions. Additionally, the authors introduce Argus, a multi-agent attribute inference framework based on abductive reasoning.

Experiments show that Argus achieves higher binary accuracy and aggregate PES scores than the baselines. Ablation studies further reveal that verification introduces a trade-off between inference binary accuracy and specificity.

Critical strengths

Overall, I like this paper a lot. I believe the benchmark makes a meaningful contribution to an important privacy problem.

  • Meaningful benchmark design and validation. The use of synthetic data informed by real social media patterns is well motivated. The benchmark also includes an adequate validation process, including human evaluations of the indistinguishability between synthetic and real posts.
  • Interesting inference framework and empirical improvements. Argus offers an interesting approach to attribute inference through abductive reasoning and demonstrates improved empirical performance. The finding that these improvements are driven by preserving and leveraging uncertain cues is particularly interesting.
  • Valuable insights into privacy risks. The analysis of how verification affects inference success and the trade-off between accuracy and specificity provides useful insights for assessing and mitigating privacy risks.
  • More nuanced evaluation metric. PES is a valuable attempt to evaluate inference specificity and information sensitivity beyond binary accuracy alone.
Critical weaknesses

My main concerns relate to the evaluation metric and the granularity of the reported results.

  1. PES does not account for false positives. Although PES rewards correct inferences based on specificity and sensitivity, it does not explicitly penalize incorrect predictions. This may favor methods that make more aggressive, low-confidence guesses over methods that infer fewer attributes more conservatively but with higher precision. Reporting PES alone therefore does not fully characterize the trade-off between inference coverage and false positives.

  2. Sensitivity modeling may not adequately capture partial disclosure. PES appears to assign a fixed sensitivity score (1–5) to each attribute. If this is the case, the metric may not fully capture how sensitivity changes with inference granularity. In particular, privacy sensitivity may vary nonlinearly between coarse-grained and highly specific disclosures.

  3. Comparability of granularity scores across attribute categories is unclear. Different attribute types, such as dates, locations, and occupations, are mapped to hierarchical structures and assigned granularity scores. However, it is unclear whether the same granularity score is comparable across categories. Since the final metric aggregates these scores, more detailed per-attribute results would help assess whether the observed improvements are consistent across categories.

Questions for authors
  1. Precision and false positives: Could the authors systematically report inference precision alongside PES for Argus and the baselines? This would provide a clearer picture of how different methods trade off inference coverage against false positives, particularly given that PES may favor aggressive guessing.

  2. Sensitivity scores and partial disclosure: Is the sensitivity component of PES a fixed score assigned to each ground-truth attribute, regardless of the granularity of the inferred information? If so, how does the metric account for the potentially nonlinear relationship between disclosure granularity and information sensitivity?

    I would appreciate two additional analyses during the rebuttal:

    • A small sensitivity analysis using alternative granularity scoring functions, such as gg, g\sqrt{g}, and g2g^2. If Argus consistently outperforms the single-agent baseline under these alternatives, this would strengthen the empirical findings without requiring the proposed formula to perfectly represent privacy harms.
    • A brief qualitative analysis of how information sensitivity varies across different levels of disclosure granularity.
  3. Cross-attribute comparability: How are granularity scores made comparable across different attribute categories, such as dates, locations, and occupations? Could the authors report per-attribute results for Argus and the baselines to show whether the performance improvements hold consistently across categories?

3 comments
AuthorsOpenReview ↗

Response to Reviewer S2rD

Dear Reviewer S2rD,

Thank you for your thoughtful review and kind words about our paper. We appreciate your recognition of the benchmark design and the value of measuring privacy exposure beyond binary accuracy.

We have uploaded the latest revised PDF. Sections, tables, and figures references below use its numbering.

W1 / Q1. How PES accounts for false positives

PES credits correct inferences according to their specificity and contextual sensitivity (Section 3.3). The denominator is the sum of the sensitivity weights of all annotated leaked attributes and is the same for every method. For an annotated leaked attribute, a missing prediction or an entirely wrong value contributes zero to the numerator. Its sensitivity weight still counts in the denominator.

Binary measures the share of annotated leaked attributes recovered at any correct level (Section 3.3). We added prediction precision and the false positive rate (FPR) to the evaluation of all six methods alongside Binary and PES (Section 5.2, Table 2) in the revised version. Prediction precision is the share of distinct reported claims that are correct at the stated level of detail. FPR is the share of user–attribute pairs annotated as unleaked for which a method reports a value.

MethodBinaryPESPrediction precisionFalse positive rate
TextLLM0.340.220.3900.433
PostVLM0.490.330.5660.296
SelfDisc0.400.280.6170.163
Holmes0.540.380.5790.317
SingleAgent0.630.440.6470.227
Argus0.710.550.8480.095

W2 / Q2.1. Sensitivity weights and alternative granularity functions

Sensitivity weights. Yes. We assign one sensitivity weight to each annotated leaked attribute (Section 3.3). The same attribute can receive different weights for different users because sensitivity depends on the value and context. Three annotators independently rescored 78 instances across all 50 users, with ordinal agreement of 0.852 (Appendix A.2.2, Table 9).

Granularity functions. To check whether Argus's lead depends on how we score partial matches, we ran a sensitivity analysis with two alternative granularity functions (Appendix E.1, Table 28). We applied each function to the granularity score before sensitivity weighting, with the predictions, sensitivity labels, and aggregation fixed.

FunctionSingleAgentArgus
g0.440.55
√g0.510.61
g²0.340.47

Argus leads SingleAgent by 0.10 to 0.13 PES under all three functions. Both alternative functions leave scores of 0 and 1 unchanged. The scores under g are the main PES results in Section 5.2 (Table 2).

Q2.2. How sensitivity varies with disclosure granularity

Harm can increase more sharply as disclosure becomes more specific. In the home-address example (Section 1 and Appendix A.2.3), a city narrows the search area, while a residential compound can make the user's home physically locatable. The resulting increase in harm can exceed the change implied by the granularity score. PES uses one sensitivity weight per instance and therefore approximates this change (Section 3.3).

W3 / Q3. Granularity score comparability and per-attribute results

Granularity scores are directly comparable only within the same attribute. In the location hierarchy in Section 3.3, a correct city earns partial credit toward a home address, and a correct full address earns more. The same granularity score for a date, an address, or an occupation can correspond to different levels of privacy harm.

To see how the gains vary by attribute, we added a comparison of all six methods on each of the 28 attributes (Appendix E.3, Table 30). The table below shows six representative attributes. Section 5.2 summarizes the gains by attribute.

AttributeTextLLMPostVLMSelfDiscHolmesSingleAgentArgus
Home Address0.200.310.250.350.390.63
Occupation0.240.360.320.410.460.64
Travel Experience0.240.380.320.420.490.66
Political Tendency0.160.280.240.340.340.41
Legal Status0.180.280.240.310.360.44
Immigration/Visa Status0.080.180.140.240.380.47

Argus ranks first among the six methods in all four attribute groups (Section 5.2 and Appendix D, Table 24). Its PES is 0.61 versus SingleAgent's 0.50 for identity, 0.56 versus 0.43 for socioeconomic status, 0.57 versus 0.46 for lifestyle and routines, and 0.52 versus 0.41 for sensitive attributes.


Thanks again for your thoughtful reviews!

Best,
Authors

Thank you for the detailed response. My main concerns are well addressed. I will increase my score.

Thank you very much for your positive review and for taking the time to review our responses!
We are glad that our responses have addressed your main concerns. We sincerely appreciate your constructive feedback and will incorporate your suggestions in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.