- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work introduces FraudSociety, a generative-agent-based framework for simulating financial fraud ecosystems. It models interactions between scammers, everyday users, and their socioeconomic surroundings. Integrating agent decision-making, social exchanges, and records of financial transactions, the system recreates how fraud spreads, how victims make decisions, and how financial losses unfold. The authors validate the simulated macro loss distribution and related metrics using real-world fraud statistics. Ablation studies are also carried out to examine the contribution of individual system components. Beyond that, the paper develops multiple anti-fraud intervention strategies. By comparing simulated losses under different intervention scenarios, it demonstrates the framework’s promise for evaluating anti-fraud policies
Critical strengths
-
This work presents a relatively complete multi-agent simulation framework for financial fraud. By integrating social interactions, agent decision-making, fraudulent activities and financial transaction records into one unified system, the paper enables investigation of the dynamic relationship between fraudulent behaviors and economic losses. This system-level integration carries certain research and practical value.
-
The framework supports practically meaningful anti-fraud intervention applications. Beyond simulating fraud activities, the paper attempts to assess how different intervention strategies affect economic losses, offering a potential experimental testbed for studying anti-fraud measures. It also includes fairly detailed descriptions of the system and experiments, which help readers understand its implementation.
Critical weaknesses
-
The simulator lacks thorough validation of its realism and carries risks of circular validation. The authors incorporate priors tied to real-world fraud data for population, economic, and fraud-type parameters when building the system. They then validate the simulator against similar macroscopic statistical patterns. This makes it hard to tell whether observed simulation outcomes emerge naturally from agent interactions, or simply stem from these pre-defined distributions. The paper also lacks quantitative comparisons against real individual behavioral traces. Even if macroscopic metrics match real-world data, this alone cannot confirm that the simulator reliably reproduces real fraud dynamics.
-
The core experiments do not sufficiently back the effectiveness of the method or conclusions on interventions. The NIA ablation only replaces the module with random action selection. It omits competitive baselines such as direct LLM-driven decision-making or well-designed rule-based strategies, so the necessity of the NIA architecture itself remains unproven. Meanwhile, the six anti-fraud intervention settings each run only a single simulation. No variance or confidence intervals are reported. Multiple intervention factors are changed simultaneously across conditions, introducing confounding effects. These flaws weaken the statistical reliability and causal interpretability of the results.
Questions for authors
-
How can the authors distinguish genuine emergent behavior from patterns imposed by the simulator's design? The simulator incorporates real-world demographic and fraud-related priors, while its validity is evaluated using similar aggregate statistics. What evidence demonstrates that the reported agreement with real-world data is not largely a consequence of these predefined assumptions?
-
What evidence supports the claim that the simulated agents realistically reproduce human fraud-related behavior? Matching aggregate fraud statistics does not necessarily imply realistic individual decision-making. Without independent behavioral validation, why should the simulator's predictions about fraud dynamics and intervention effectiveness be considered reliable?
-
What is the specific contribution of the proposed NIA mechanism beyond simply avoiding random decisions? The current ablation replaces NIA with random action selection, which does not establish the necessity of its proposed structure. What evidence shows that NIA provides meaningful advantages over standard LLM-based decision-making or simpler rule-based policies?
5 comments
W2 and Q3: Role of the NIA Architecture and Controlled Multi-Run Intervention Experiments
Thank you for asking what specific role NIA plays beyond avoiding random actions, and for emphasizing multi-run confidence intervals and single-factor isolation in our intervention experiments. Below, we clarify how our NIA ablation works on the same reference setting, report 10-run confidence intervals across all warnings, and isolate warning content, timing, and delivery channel.
1. What w/o NIA Actually Tests: Need-Conditioned Routing vs. Uniform Routing With LLM Action Selection.
We clarify how w/o NIA in Table 3b and Table 8 is implemented:
w/o NIAdoes NOT replace concrete action selection with random actions. Once an activity branch is selected, every concrete social, consumption, and payment action is still chosen byQwen3-14Bfrom the exact same bounded observation.- What
w/o NIAreplaces is need-conditioned intent routing with uniform route sampling across feasible activity branches:
| Model Variant on Table 3b Reference Setting | Action Policy | Route Selection | HTL↓ | ALS↓ | FSH↓ | ALR↓ | Composite QD↓ | QD Change vs. w/o NIA |
|---|---|---|---|---|---|---|---|---|
| w/o NIA in Table 3b | Qwen3-14B LLM | Uniform Route | 1.0000 | 0.6538 | 0.7908 | 0.1883 | 0.6582 | +0.0% Reference Baseline |
| w/o Ledger in Table 3b | Qwen3-14B LLM | Need-Conditioned | 0.7005 | 0.5726 | 0.3410 | 0.1898 | 0.4510 | −31.5% vs. w/o NIA |
| Full Model in Table 3b | Qwen3-14B LLM | Need-Conditioned | 0.0893 | 0.6513 | 0.3012 | 0.1495 | 0.2978 | −54.8% vs. w/o NIA |
In FraudSociety, working, nearby consumption, and social media have separate action spaces and execution gates. NIA links financial scarcity and unmet social needs to when civilians enter social interactions—where civilians under high safety need exhibit 1.94 vs. 0.94 transfers per 100h in Table 3a.
Our ablation demonstrates that NIA goes well beyond avoiding random actions: even when the same LLM selects every concrete action, need-conditioned routing is essential for connecting financial stress and social needs to scam exposure, reproducing realistic heavy-tailed losses and fraud-family severity.
2. 10-Run Paired Confidence Intervals Across All Intervention Conditions.
We expanded all interventions to 10 paired runs covering 1,120 initial civilians, and added a realistic Detector-Driven Warning Policy observing only visible messages and the user's own payment history—operating at 73.9% Precision, 68.0% Recall, 6.0% FPR, and 2h median delay while reducing normal payments by 2.1%. Across the 10 runs, C5 beats C4 in 8 of 10 runs and C1 and C2 swap ranks in 3 of 10 runs:
| Condition across 10 Paired Runs of 1,120 Civilians | Mean Loss Red. vs. C0 | 95% CI across 10 Runs | Cohort Repeat Civilians | Fixed-Cohort Repeat Share |
|---|---|---|---|---|
| C0: No Warning Baseline | 0.0% Ref. | 0.0% to 0.0% | 346/1,120 | 30.9% |
| C1: Platform Broadcast | 30.7% | 22.8% to 38.6% | 243/1,120 | 21.7% |
| C2: Monitor Labels | 39.9% | 33.3% to 46.5% | 272/1,120 | 24.3% |
| C3: Random Comments | 64.0% | 57.6% to 70.4% | 196/1,120 | 17.5% |
| C4: Peer Warnings | 76.2% | 71.6% to 80.8% | 136/1,120 | 12.1% |
| Detector-Driven Realistic | 59.8% | 54.3% to 65.3% | 194/1,120 | 17.3%, Δ −13.6 pp |
| C5: Oracle Personalized | 82.6% | 79.0% to 86.2% | 78/1,120 | 7.0% |
Across 10 paired seeds, confidence intervals remain tight and the overall ranking of warning strategies stays consistent—confirming that our intervention conclusions are statistically robust and hold under a realistic non-oracle detector.
3. Single-Factor Isolation of Content, Timing, and Channel.
To separate the dimensions that co-vary in C1–C5, we fixed the exact same detection triggers and warning opportunities across 10 paired runs and varied only one factor at a time:
| Isolated Factor across 10 Runs | Baseline Level | Target Level | Marginal Gain | 95% CI | Cohort Repeat Shift |
|---|---|---|---|---|---|
| Content: Contextual vs. Generic | Generic 47.0% | Contextual 59.8% | +12.8 pp | +8.4 to +17.2 pp | 23.4% → 17.3%, −6.1 pp |
| Timing: Early vs. Delayed | Delayed 52.8% | Early 59.8% | +7.0 pp | +4.3 to +9.7 pp | 20.8% → 17.3%, −3.5 pp |
| Channel: Private DM vs. Public | Public 56.6% | Private DM 59.8% | +3.2 pp | −1.8 to +8.2 pp | 18.6% → 17.3%, −1.3 pp |
Isolating each dimension under matched detection triggers shows that context-specific warning content and early trigger timing deliver the primary marginal gains in fraud reduction, whereas switching between public comments and private DMs shows a smaller, statistically uncertain difference.
We hope the above clarifications and the additional experiments sufficiently addressed your concerns. If you are satisfied, we kindly request you to consider updating the score to reflect the newly added results and discussion. We remain committed to addressing any remaining points you may have during the discussion phase.
W1, Q1, and Q2: Disentangling Priors from Interaction Dynamics, Individual Realism, and Intervention Validation
Thank you for asking how we know our findings emerge from agent interactions rather than preset priors, and what independent evidence validates individual agent behavior. Below, we compare prior removal against feedback removal across 10 runs, and report blind human audits of all 81 victim pathways and 250 pre/post-warning trajectory pairs across C1–C5.
1. Are Macro Patterns Just Baked into the Priors, or Do They Emerge from Agent Interactions?
Across 10 runs per condition, we compare removing age and fraud-family amount guidance against keeping all priors intact while disabling closed-loop financial feedback or running a shuffled diagnostic that preserves per-transfer amounts Pv,c,t while reassigning transfers across contacted civilians—so follow-up transfers no longer accumulate on the same victims into totals Lv=∑c,tPv,c,t:
Comparing prior removal against feedback removal shows that preset priors alone cannot account for our results: removing amount guidance shifts QD modestly, whereas breaking closed-loop feedback or victim-level transfer accumulation breaks heavy-tailed loss alignment, proving macro patterns emerge from multi-step interactions.
2. Independent Validation of Individual Victim Behavior.
Three raters blindly evaluated all 81 positive-net-loss victim trajectories, inspecting posts, private chats, payments, bait returns, and withdrawal requests. Raters judged most pathways across every scam family realistic with substantial agreement, while the 13 unapproved cases mainly showed rapid trust progression or repetitive payment excuses:
Blind human review of all victim trajectories confirms that individual civilians exhibit coherent, realistic decision-making across scam families rather than merely echoing macro priors.
3. Intervention Validation: 10-Expert Audit.
To test whether warning responses match human behavior, 10 anti-fraud practitioners blindly classified 250 pre/post-warning trajectory pairs—50 per strategy across
C1–C5—as Real or Fake, totaling2,500judgments. Overall,2,029of2,500ratings, or81.2%, judged pairs Real vs.471Fake, with individual experts ranging from78.0%at195/250to84.8%at212/250. By majority vote,215/250or86.0%of pairs were judged Real, with33Fake and25:5ties inC3andC4:10-expert blind validation of warning pairs confirms that simulated interventions rest on realistic human responses, supporting pre-deployment policy evaluation.