- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates cumulative privacy leakage from public social media posts, where seemingly harmless textual and visual clues can be combined to infer private user attributes. The authors make three main contributions: (1) SopriBench, a synthetic benchmark containing 50 user profiles, 500 posts, and 1,569 images, with annotations for private attributes, supporting evidence, and leakage characteristics; (2) Privacy Exposure Score (PES), a metric that evaluates inferred attributes based on their granularity and contextual sensitivity; and (3) Argus, a training-free agentic framework that maintains hypotheses, aggregates cross-post evidence through a graph structure, and verifies inferences using external tools.
Experiments show that Argus achieves a 25% relative PES improvement over the strongest baseline, SingleAgent, with particularly notable gains on cross-post leakage. The authors further provide ablation studies, human annotation audits, and a small-scale real-user validation involving 11 participants to assess the framework's effectiveness and evaluation reliability.
Critical strengths
1. In-depth Privacy Scenario. The paper provides a thorough investigation of privacy leakage in public social media, moving beyond isolated disclosures to examine how seemingly harmless clues accumulate across posts, modalities, and contextual information. I particularly appreciate the authors' effort to characterize this specific scenario in depth, including different leakage mechanisms, inference difficulties, and contextual sensitivity. This focused treatment makes the problem setting meaningful and relevant to real-world privacy risks.
2. Substantial Research Effort. The amount of work invested in this study is commendable. The authors analyze a substantial real-user reference corpus, construct a multimodal synthetic benchmark with detailed annotations, and develop a multi-stage agentic framework. They also conduct baseline comparisons, ablation studies, human annotation audits, and real-user validation. These efforts strengthen the credibility of the study.
3. Well-motivated Agentic Framework. Argus explicitly maintains unresolved hypotheses, collects additional evidence through external tools, and verifies inferences before incorporating them into user profiles. This design is well aligned with the cumulative nature of privacy leakage. The improvements on cross-post inference tasks provide empirical support for the value of structured evidence management and iterative verification.
Critical weaknesses
-
[Major] Incomplete Case Study. Although the paper provides extensive methodological details and a qualitative example in Figure 3, I would appreciate a complete, concrete case study illustrating the entire pipeline. Showing the original multimodal posts, ground-truth attributes, extracted evidence, hypothesis updates, tool outputs, verification decisions, and final privacy profile would make the framework easier to understand and assess. Given the complexity of the system, I consider such an end-to-end example important.
-
[Minor] Complexity and LLM Dependence. The framework involves numerous components, models, and LLM-based decisions throughout data construction, inference, verification, and evaluation. While this is not necessarily a methodological flaw, I am concerned that the overall design may be unnecessarily complicated and introduce multiple sources of model-dependent bias. Despite the human evaluation audits, the robustness of the conclusions to different model choices, prompts, and evaluation procedures remains insufficiently explored.
-
[Minor] Synthetic Benchmark Validity. SopriBench is carefully constructed, but its privacy leakage patterns are explicitly designed through predefined user profiles and leakage plans. This raises concerns about whether recovering these planned attributes adequately reflects the uncertainty and ambiguity of real-world privacy inference. Although the real-user validation is encouraging, its limited scale makes it difficult to fully establish the benchmark's generalizability.
-
[Minor] Insufficiently Controlled Comparisons. Argus substantially outperforms SingleAgent, but it also uses considerably more model calls, tool calls, and execution time. Therefore, it is difficult to isolate how much of the improvement comes from the proposed reasoning framework rather than additional computational effort. Moreover, the CPeg ablation compares graph-based memory against conversation history rather than an equally informative structured-memory baseline. More carefully controlled comparisons would strengthen the claims about the framework's effectiveness and novelty.
-
[Minor] Subjectivity in Agent Design. Several core components of Argus appear to rely on heuristic or LLM-driven decisions, including hypothesis prioritization, tool routing, evidence verification, and hypothesis acceptance. Although the authors describe the general principles and fix the routing policy before evaluation, the operational criteria behind these decisions remain insufficiently specified. In particular, it is unclear how confidence scores are determined, what constitutes sufficient evidence, and how subjective decisions affect the final results. I would appreciate clearer design justifications and analyses demonstrating the robustness of these choices.
Questions for authors
My main questions and concerns are detailed in the Weaknesses section.
I appreciate the authors' in-depth investigation of social media privacy leakage and the substantial effort invested in this work. However, my primary concern is the lack of a complete end-to-end case study, which makes it difficult to fully understand and assess the proposed framework. If the authors can provide one or more concrete examples illustrating the complete reasoning and verification process, I would be happy to reconsider my assessment and raise my score.
8 comments
Response to Reviewer 2oCp (2/3)
W3. Real-world validity of the synthetic benchmark
The original submission describes how we use patterns from 318 reference accounts to guide benchmark construction and how the leakage plans specify target attributes and clues (Sections 3.1–3.2). It also reports a 104-response study of profile authenticity and image discrimination (Appendix A.3).
To check what the synthetic posts reveal, we added a comparison of clue patterns and an audit of support for the annotated attributes in the revision (Section 3.2 and Appendix A.3.1). We used the same clue carrier and cross-post relation categories for the reference and synthetic accounts. In the audit, we checked whether the synthetic cues were visible, consistent across posts, and sufficient to support the annotated value.
| Check | Result |
|---|---|
| Synthetic age, location, occupation, and education | Every synthetic ground-truth profile contains all four fields. |
| Reference versus synthetic cue carriers and cross-post relations | The overlap is 0.90 for clue carriers and 0.87 for cross-post relations. |
| Synthetic cue visibility, consistency, and support | Visibility is 94.0% (470/500 posts), consistency is 91.6% (458/500 posts), and full cross-post support is 95.7% (379/396 D4 attributes). |
To check the account mix, we also compared age and work or study status across the reference and synthetic accounts (Appendix A.3.1, Table 14). We used ten posts per account and recorded only explicitly stated information. Missing or ambiguous values are marked Unknown. Employed accounts make up 52.0% of the synthetic set, compared with 42.2% of real Rednote and 45.2% of real Instagram. Unknown work or study status accounts for 4.0%, compared with 15.5% and 12.7%. All age and work/study categories are represented.
We then compared how often clues about each type of private information appear in the two sets of accounts. We use real accounts for aggregate checks and synthetic data for ground-truth evaluation. The table reports the share of accounts with at least one clue in each privacy category.
| Privacy category | Real reference share (%) | Synthetic share (%) |
|---|---|---|
| Identity documents or ID-related information | 21.7% | 28.0% |
| Home address | 56.6% | 70.0% |
| Occupation and workplace | 65.7% | 82.0% |
| Education and study history | 48.7% | 58.0% |
| Income, finance, and assets | 42.8% | 46.0% |
| Travel and daily routines | 69.5% | 76.0% |
We also ran a questionnaire to study unintended disclosure in everyday sharing. We examined which privacy clues participants might overlook when deciding what to share and what they could infer from the same synthetic posts. We collected 19 responses (Appendix A.3.1). We asked participants which posts they would share on a public account and which details they would change, crop, or omit, with their reasons. Participants rated typicality on a 1–5 scale, flagged details that seemed inconsistent or deliberately placed, and gave the most specific occupation they could infer with supporting post IDs.
| Questionnaire measure | Result |
|---|---|
| Post as-is | 26.3% |
| Edit or omit content | 68.4% |
| Mean typicality score (1–5) | 4.2 |
| Inconsistency-flag rate | 10.5% |
| Correct occupation-inference rate | 84.2% |
| Correct supporting-post identification rate | 78.9% |
| Deliberately-placed-detail flag rate | 15.8% |
| Other / do not post or unsure | 5.3% |
To check whether the gains extend across user groups, we added a performance breakdown by work or study status (Appendix D, Table 25).
| Group | Share (%) | Attributes | SingleAgent PES | Argus PES | SingleAgent cross-post PES | Argus cross-post PES |
|---|---|---|---|---|---|---|
| Student | 20.0% | 156 | 0.45 | 0.56 | 0.40 | 0.53 |
| Employed | 52.0% | 406 | 0.43 | 0.54 | 0.38 | 0.50 |
| Self-employed | 18.0% | 140 | 0.45 | 0.56 | 0.41 | 0.53 |
| Retired | 6.0% | 47 | — | — | — | — |
| Unknown | 4.0% | 32 | — | — | — | — |
| Total | 100.0% | 781 | 0.44 | 0.55 | 0.39 | 0.51 |
Argus leads in all three groups with at least eight accounts. Employed is the largest group and has the lowest SingleAgent PES of the three groups (0.43). Retired and Unknown are not interpreted separately because each has fewer than eight accounts.
In the 11-user validation reported in the original submission (Section 5.2 and Appendix C, Table 19), Argus reaches 75.5% confirmed precision and 11.2 confirmed attributes per user, compared with 51.4% and 9.9 for SingleAgent. This provides a direct check on those accounts alongside the controlled synthetic results. Thank you again for specifying the evidence needed to assess the framework.
Response to Reviewer 2oCp (3/3)
W4. Budget matching and structured memory baselines
Resource budgets. The original submission reports full-run costs (Appendix B.7, Table 15). To test whether Argus's gains persist at comparable resource use, we added comparisons at separate token, tool-call, and time limits in revised Section 5.4 (Table 5). For each user, SingleAgent decides when to stop, and its completed run sets separate limits for tokens, tool calls, and time. The paired runs use the same model, posts, and tools. For each limit, we build the Argus profile from evidence accepted by the last investigation round completed within the limit. We measure time from the start of each run to SingleAgent's final profile or the end of the selected Argus round. Token counts include model input and output. We score each profile as in the main comparison, reuse saved judgments for unchanged scoring inputs, and assign zero to omitted attributes.
Argus scores higher at all three limits. At the time limit, it averages 4.7 minutes per user compared with 4.8 for SingleAgent, with gains of 0.02 overall PES and 0.03 cross-post PES. Tokens, tool calls, and time in the table are per-user averages.
| Method | Tokens | Tool calls | Time (min) | PES | Cross-post PES |
|---|---|---|---|---|---|
| SingleAgent | 86.5K | 5.1 | 4.8 | 0.44 | 0.39 |
| Argus (tokens) | 85.7K | 6.3 | 6.5 | 0.48 | 0.45 |
| Argus (tool calls) | 82.6K | 5.1 | 6.2 | 0.47 | 0.44 |
| Argus (time) | 81.9K | 4.8 | 4.7 | 0.46 | 0.42 |
Revised Appendix B.7 describes the replay and scoring protocol.
The original submission reports the ablation that replaces CPeg with conversation history (Section 5.4, Table 3). Revised Section 5.3 (Table 3) reports a drop in cross-post PES from 0.51 to 0.38.
To test the effect of evidence organization, we compared CPeg with a structured ledger in an additional experiment (Section 5.3 and Appendix E.2). Both received the same saved observations, provenance, hypotheses, and support links. We kept the model, verification criteria, and context allowance fixed. The ledger yields 0.50 PES, 0.42 cross-post PES, 74.5% precision, and 104.0K tokens per user. The corresponding CPeg results are 0.55, 0.51, 84.8%, and 98.6K.
W5. Hypothesis prioritization, tool routing, and evidence requirements
Hypothesis prioritization, routing, and admission are governed by explicit criteria. At each round, Argus first investigates the highest-priority unresolved hypothesis. The original submission describes the hypothesis store, routing policy, and evidence verification in Appendices B.2–B.4. The router considers missing evidence, the likely value of a route, the remaining budget, and prior attempts. It uses fixed rules for common routes and an LLM fallback when no rule matches. Admission is a separate decision about whether the source is reliable and the chain supports the claimed value at the stated specificity. For example, identifying the compound in a photo establishes its location. A claim that the user lives there requires evidence that links the user to the compound.
In the original Verifier ablation (Section 5.4, Table 3), removing the Verifier raises coverage from 0.71 to 0.74, lowers granularity from 0.60 to 0.57, and lowers PES from 0.55 to 0.50.
To see how the results depend on these design choices, we added an experiment that changes one investigation rule at a time (Appendix E.2, Table 29). We tested alternatives for hypothesis ordering, routing fallback, and evidence admission, with inputs and resource limits fixed.
| Variant | PES | Precision (%) | False positives/user | Tokens/user |
|---|---|---|---|---|
| Default | 0.55 | 84.8 | 1.16 | 117.9K |
| Fixed hypothesis order | 0.50 | 79.5 | 2.20 | 116.0K |
| Fixed tool selection rules | 0.47 | 76.3 | 2.84 | 109.1K |
| Independent evidence required | 0.53 | 87.8 | 1.20 | 125.7K |
Thanks again for your thoughtful reviews and your time and effort to our community!
Best,
Authors
Response to Reviewer 2oCp (1/3)
Dear Reviewer 2oCp,
Thank you for your detailed comments on the major and minor points and for recognizing our investigation of cumulative leakage and the motivation for evidence management.
We have uploaded the latest revised PDF. References use the revised numbering unless we specify the original submission.
W1. End-to-end case study of Argus
Figure 3 of the original submission illustrates residence inference. To show the full process, we added a complete end-to-end case study of Argus and SingleAgent in revised Section 5.2 and Appendix D.1. It includes post excerpts, ground-truth attributes, extracted evidence, hypothesis updates, tool outputs, verification decisions, and final privacy profiles. Figure 3 compares the two methods at successive steps of the home and workplace investigations. It uses redacted source images, including a magnified crop of the vehicle app's location field. The venue map, notice, outing photo, and folder crest show the recorded cues.
The case shows how Argus keeps address candidates open, keeps parking and qualification clues separate from employment, and checks later evidence before accepting an early guess.
Appendix D.1 gives the annotated values and lists all nine post excerpts in Table 27. It records the tool observations, the first incorrect choices, how each method treats the later evidence, and the final profiles.
W2. System complexity and evaluator choice
Figure 2 of the original submission shows the initial skim, targeted investigation, and evidence verification. The same submission reports 0.7 deep visual inspections out of 13.3 routed actions per user (Appendix B.7, Table 17) and describes human checks for cue plausibility and unintended identifiers (Appendix A.3). In its sensitivity audit, three annotators independently rescored 78 of 781 leaked attributes, with ordinal agreement of 0.852 (Appendix A.2.2, Table 7). Its evaluator audit shows that the six-method ranking also holds with adjudicated human labels (Appendix C).
To check whether Argus's lead depends on the evaluator, we added a comparison with two alternative models in revised Section 5.3 (Table 4). We rescored the 546 decisions on 78 leaked attributes from the original human audit (Appendix C) with DeepSeek-V4.1-Flash and GLM-5.3-Flash. Both received the same predictions, annotations, and scoring rules as GPT-5.5, with method names withheld. We compared their slot matches and granularity scores with the adjudicated human labels and computed both methods' PES on the same subset.
The Argus–SingleAgent PES gaps under the two evaluators are 0.11 and 0.09, respectively. Appendix C of the original submission describes the main scoring procedure.
The original submission reports a shared-model comparison of Argus and SingleAgent (Section 5.2 and Appendix B.7, Table 16). With Qwen3.6-35B-A3B, Argus scores 0.50 PES compared with 0.39 for SingleAgent. To control for model choice across methods, we extended this comparison to all six methods in revised Appendix E.2.
We ran a separate experiment to see whether Argus's lead depends on the wording of the prompts (Appendix E.2). We changed one prompt setting at a time, with the same inputs and resource limits. Argus scores 0.53–0.56 PES, compared with 0.43–0.45 for SingleAgent, with paired gaps of +0.10 to +0.12. Argus leads SingleAgent in each prompt setting.