acceptodds
Desk rejected at ICLR 2027

FraudSociety: A Generative Agent-Based Society for the Financial-Fraud Economy

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Financial fraud caused over $15.9 billion in losses in the United States in 2025, according to the Federal Trade Commission. Understanding such large-scale harm requires jointly modeling informational and financial dynamics. Yet existing simulators typically capture only one side of this interaction, leaving a gap in modeling how fraud unfolds in the real world. To bridge this gap, we introduce FraudSociety, the first LLM-agent-based simulator that jointly models these coupled dynamics. It integrates a dynamic social environment for information exposure and communication, persistent financial ledgers that preserve payment and loss histories, and a Need-Intent-Action architecture for civilian decision making. Fraudsters further adapt their communication and financial actions to individual interaction histories and ledger states, forming a closed feedback loop between the two dynamics. We validate FraudSociety across macro-level outcomes, micro-level behavior, and intervention effects. At the macro level, the simulated fraud outcomes closely match real-world patterns across multiple key statistics. At the micro level, it reproduces strong loss concentration, with the top 10% of victimization records accounting for 81.1% of total losses, consistent with Canadian Anti-Fraud Centre statistics. Counter-fraud experiments further reveal variation across intervention strategies, with the best-performing personalized intervention reducing the repeat-payment rate from 49.4% to 10.9%. These results establish \textscFraudSociety as both a high-fidelity simulator and a scalable testbed for evaluating anti-fraud strategies under diverse intervention conditions, thereby advancing research on financial fraud and its prevention.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Financial Fraud, LLM-Based Agents, Agent-Based Social Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6ARkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces FraudSociety, a generative-agent-based framework for simulating financial fraud ecosystems. It models interactions between scammers, everyday users, and their socioeconomic surroundings. Integrating agent decision-making, social exchanges, and records of financial transactions, the system recreates how fraud spreads, how victims make decisions, and how financial losses unfold. The authors validate the simulated macro loss distribution and related metrics using real-world fraud statistics. Ablation studies are also carried out to examine the contribution of individual system components. Beyond that, the paper develops multiple anti-fraud intervention strategies. By comparing simulated losses under different intervention scenarios, it demonstrates the framework’s promise for evaluating anti-fraud policies

Critical strengths
  1. This work presents a relatively complete multi-agent simulation framework for financial fraud. By integrating social interactions, agent decision-making, fraudulent activities and financial transaction records into one unified system, the paper enables investigation of the dynamic relationship between fraudulent behaviors and economic losses. This system-level integration carries certain research and practical value.

  2. The framework supports practically meaningful anti-fraud intervention applications. Beyond simulating fraud activities, the paper attempts to assess how different intervention strategies affect economic losses, offering a potential experimental testbed for studying anti-fraud measures. It also includes fairly detailed descriptions of the system and experiments, which help readers understand its implementation.

Critical weaknesses
  1. The simulator lacks thorough validation of its realism and carries risks of circular validation. The authors incorporate priors tied to real-world fraud data for population, economic, and fraud-type parameters when building the system. They then validate the simulator against similar macroscopic statistical patterns. This makes it hard to tell whether observed simulation outcomes emerge naturally from agent interactions, or simply stem from these pre-defined distributions. The paper also lacks quantitative comparisons against real individual behavioral traces. Even if macroscopic metrics match real-world data, this alone cannot confirm that the simulator reliably reproduces real fraud dynamics.

  2. The core experiments do not sufficiently back the effectiveness of the method or conclusions on interventions. The NIA ablation only replaces the module with random action selection. It omits competitive baselines such as direct LLM-driven decision-making or well-designed rule-based strategies, so the necessity of the NIA architecture itself remains unproven. Meanwhile, the six anti-fraud intervention settings each run only a single simulation. No variance or confidence intervals are reported. Multiple intervention factors are changed simultaneously across conditions, introducing confounding effects. These flaws weaken the statistical reliability and causal interpretability of the results.

Questions for authors
  1. How can the authors distinguish genuine emergent behavior from patterns imposed by the simulator's design? The simulator incorporates real-world demographic and fraud-related priors, while its validity is evaluated using similar aggregate statistics. What evidence demonstrates that the reported agreement with real-world data is not largely a consequence of these predefined assumptions?

  2. What evidence supports the claim that the simulated agents realistically reproduce human fraud-related behavior? Matching aggregate fraud statistics does not necessarily imply realistic individual decision-making. Without independent behavioral validation, why should the simulator's predictions about fraud dynamics and intervention effectiveness be considered reliable?

  3. What is the specific contribution of the proposed NIA mechanism beyond simply avoiding random decisions? The current ablation replaces NIA with random action selection, which does not establish the necessity of its proposed structure. What evidence shows that NIA provides meaningful advantages over standard LLM-based decision-making or simpler rule-based policies?

5 comments

W1, Q1, and Q2: Disentangling Priors from Interaction Dynamics, Individual Realism, and Intervention Validation

Thank you for asking how we know our findings emerge from agent interactions rather than preset priors, and what independent evidence validates individual agent behavior. Below, we compare prior removal against feedback removal across 10 runs, and report blind human audits of all 81 victim pathways and 250 pre/post-warning trajectory pairs across C1–C5.

1. Are Macro Patterns Just Baked into the Priors, or Do They Emerge from Agent Interactions?

Across 10 runs per condition, we compare removing age and fraud-family amount guidance against keeping all priors intact while disabling closed-loop financial feedback or running a shuffled diagnostic that preserves per-transfer amounts Pv,c,tP_{v,c,t} while reassigning transfers across contacted civilians—so follow-up transfers no longer accumulate on the same victims into totals Lv=∑c,tPv,c,tL_v = \sum_{c,t} P_{v,c,t}:

Model, Prior, or Feedback Variant across 10 RunsHTL↓ALS↓FSH↓ALR↓Mean QD±SD↓Δ QD vs. Full
Full Model Baseline0.11200.66200.34700.17420.3238±0.0395+0.0000
w/o Age Amount Guidance0.14500.72400.35800.17900.3515±0.0514+0.0277
w/o Fraud-Family Amount Guidance0.12900.67000.44000.17900.3545±0.0554+0.0307
w/o Both Amount Guidances0.16500.74800.48200.18700.3955±0.0633+0.0717
w/o Civilian Financial Feedback0.60400.59200.36500.20400.4413±0.0672+0.1175
Matched Per-Transfer Marginals, Shuffled0.52600.70700.45100.21800.4755±0.0712+0.1517

Comparing prior removal against feedback removal shows that preset priors alone cannot account for our results: removing amount guidance shifts QD modestly, whereas breaking closed-loop feedback or victim-level transfer accumulation breaks heavy-tailed loss alignment, proving macro patterns emerge from multi-step interactions.

2. Independent Validation of Individual Victim Behavior.

Three raters blindly evaluated all 81 positive-net-loss victim trajectories, inspecting posts, private chats, payments, bait returns, and withdrawal requests. Raters judged most pathways across every scam family realistic with substantial agreement, while the 13 unapproved cases mainly showed rapid trust progression or repetitive payment excuses:

Fraud FamilyRecordsVictimsNet Loss ShareLagApproved ≥2/3Split 3/2/1/0
Investment Fraud191861.6%, $89.4K114h16/19, 84.2%14/2/1/2
Impersonation Fraud23208.0%, $11.6K50h20/23, 87.0%17/3/0/3
Marketplace Fraud343127.9%, $40.5K52h28/34, 82.4%24/4/1/5
Romance Fraud552.5%, $3.6K203h4/5, 80.0%3/1/0/1
All Pathways Combined8163100.0%, $145.0K72h68/81, 84.0%58/10/2/11, κ=0.68

Blind human review of all victim trajectories confirms that individual civilians exhibit coherent, realistic decision-making across scam families rather than merely echoing macro priors.

3. Intervention Validation: 10-Expert Audit.

To test whether warning responses match human behavior, 10 anti-fraud practitioners blindly classified 250 pre/post-warning trajectory pairs—50 per strategy across C1–C5—as Real or Fake, totaling 2,500 judgments. Overall, 2,029 of 2,500 ratings, or 81.2%, judged pairs Real vs. 471 Fake, with individual experts ranging from 78.0% at 195/250 to 84.8% at 212/250. By majority vote, 215/250 or 86.0% of pairs were judged Real, with 33 Fake and 2 5:5 ties in C3 and C4:

Strategy, 50 Pairs EachRealFakeReal ShareMajority Approved ≥6/10
C1: Platform Broadcast4217984.2%46/50, 92.0%
C2: Monitor Labels4168483.2%44/50, 88.0%
C3: Random Comments39011078.0%41/50, 82.0%
C4: Peer Warnings4118982.2%44/50, 88.0%
C5: Oracle Personalized39110978.2%40/50, 80.0%
Overall, 2,500 Judgments2,02947181.2%215/250, 86.0%

10-expert blind validation of warning pairs confirms that simulated interventions rest on realistic human responses, supporting pre-deployment policy evaluation.

W2 and Q3: Role of the NIA Architecture and Controlled Multi-Run Intervention Experiments

Thank you for asking what specific role NIA plays beyond avoiding random actions, and for emphasizing multi-run confidence intervals and single-factor isolation in our intervention experiments. Below, we clarify how our NIA ablation works on the same reference setting, report 10-run confidence intervals across all warnings, and isolate warning content, timing, and delivery channel.

1. What w/o NIA Actually Tests: Need-Conditioned Routing vs. Uniform Routing With LLM Action Selection.

We clarify how w/o NIA in Table 3b and Table 8 is implemented:

  • w/o NIA does NOT replace concrete action selection with random actions. Once an activity branch is selected, every concrete social, consumption, and payment action is still chosen by Qwen3-14B from the exact same bounded observation.
  • What w/o NIA replaces is need-conditioned intent routing with uniform route sampling across feasible activity branches:
Model Variant on Table 3b Reference SettingAction PolicyRoute SelectionHTL↓ALS↓FSH↓ALR↓Composite QD↓QD Change vs. w/o NIA
w/o NIA in Table 3bQwen3-14B LLMUniform Route1.00000.65380.79080.18830.6582+0.0% Reference Baseline
w/o Ledger in Table 3bQwen3-14B LLMNeed-Conditioned0.70050.57260.34100.18980.4510−31.5% vs. w/o NIA
Full Model in Table 3bQwen3-14B LLMNeed-Conditioned0.08930.65130.30120.14950.2978−54.8% vs. w/o NIA

In FraudSociety, working, nearby consumption, and social media have separate action spaces and execution gates. NIA links financial scarcity and unmet social needs to when civilians enter social interactions—where civilians under high safety need exhibit 1.94 vs. 0.94 transfers per 100h in Table 3a.

Our ablation demonstrates that NIA goes well beyond avoiding random actions: even when the same LLM selects every concrete action, need-conditioned routing is essential for connecting financial stress and social needs to scam exposure, reproducing realistic heavy-tailed losses and fraud-family severity.

2. 10-Run Paired Confidence Intervals Across All Intervention Conditions.

We expanded all interventions to 10 paired runs covering 1,120 initial civilians, and added a realistic Detector-Driven Warning Policy observing only visible messages and the user's own payment history—operating at 73.9% Precision, 68.0% Recall, 6.0% FPR, and 2h median delay while reducing normal payments by 2.1%. Across the 10 runs, C5 beats C4 in 8 of 10 runs and C1 and C2 swap ranks in 3 of 10 runs:

Condition across 10 Paired Runs of 1,120 CiviliansMean Loss Red. vs. C095% CI across 10 RunsCohort Repeat CiviliansFixed-Cohort Repeat Share
C0: No Warning Baseline0.0% Ref.0.0% to 0.0%346/1,12030.9%
C1: Platform Broadcast30.7%22.8% to 38.6%243/1,12021.7%
C2: Monitor Labels39.9%33.3% to 46.5%272/1,12024.3%
C3: Random Comments64.0%57.6% to 70.4%196/1,12017.5%
C4: Peer Warnings76.2%71.6% to 80.8%136/1,12012.1%
Detector-Driven Realistic59.8%54.3% to 65.3%194/1,12017.3%, Δ −13.6 pp
C5: Oracle Personalized82.6%79.0% to 86.2%78/1,1207.0%

Across 10 paired seeds, confidence intervals remain tight and the overall ranking of warning strategies stays consistent—confirming that our intervention conclusions are statistically robust and hold under a realistic non-oracle detector.

3. Single-Factor Isolation of Content, Timing, and Channel.

To separate the dimensions that co-vary in C1–C5, we fixed the exact same detection triggers and warning opportunities across 10 paired runs and varied only one factor at a time:

Isolated Factor across 10 RunsBaseline LevelTarget LevelMarginal Gain95% CICohort Repeat Shift
Content: Contextual vs. GenericGeneric 47.0%Contextual 59.8%+12.8 pp+8.4 to +17.2 pp23.4% → 17.3%, −6.1 pp
Timing: Early vs. DelayedDelayed 52.8%Early 59.8%+7.0 pp+4.3 to +9.7 pp20.8% → 17.3%, −3.5 pp
Channel: Private DM vs. PublicPublic 56.6%Private DM 59.8%+3.2 pp−1.8 to +8.2 pp18.6% → 17.3%, −1.3 pp

Isolating each dimension under matched detection triggers shows that context-specific warning content and early trigger timing deliver the primary marginal gains in fraud reduction, whereas switching between public comments and private DMs shows a smaller, statistically uncertain difference.

We hope the above clarifications and the additional experiments sufficiently addressed your concerns. If you are satisfied, we kindly request you to consider updating the score to reflect the newly added results and discussion. We remain committed to addressing any remaining points you may have during the discussion phase.

Reviewer_uaJkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FRAUDSOCIETY, an LLM-agent-based simulator that jointly models social interactions and financial transactions in fraud scenarios. It combines a Need-Intent-Action (NIA) framework for civilian behavior, adaptive fraudster agents, a dynamic social environment, and persistent financial ledgers. The evaluation examines alignment with aggregate fraud statistics, simulated victimization pathways, and the effects of counter-fraud interventions. However, the novelty of the behavioral framework and the validity of the simulation results remain insufficiently established.

Critical strengths
  • Integrated simulation environment. The paper connects information exposure, social interactions, financial transactions, and persistent financial states, providing a more comprehensive simulation of fraud processes than approaches focused solely on communication or transactions.

  • Multi-level evaluation design. The paper examines aggregate loss patterns, individual victimization pathways, and counter-fraud interventions. The mechanism ablations also provide preliminary evidence regarding the contributions of NIA and ledger feedback to the simulator's aggregate statistical alignment.

Critical weaknesses
  • The Need-Intent-Action (NIA) mechanism is essentially a general-purpose agent behavior modeling paradigm that decomposes decision-making into needs, intent selection, and action generation, rather than a fraud-specific behavioral model. The paper does not sufficiently establish what distinctive behavioral mechanisms NIA captures beyond this generic formulation or why it is particularly necessary for financial-fraud simulation. The contribution therefore appears to rely more on integrating existing agent-simulation components than on a substantively novel behavioral model.

  • The evaluation metrics primarily measure alignment with selected aggregate fraud statistics and changes in simulated payment behavior, rather than directly establishing whether the simulator reliably reproduces real-world victimization mechanisms or supports effective fraud prevention. The Quartet Distance (QD) is constructed from a limited set of stylized statistical patterns, and matching these patterns does not necessarily validate the underlying behavioral processes. Similarly, the intervention experiments report changes in simulated losses and payment rates without validation against observed intervention outcomes. Consequently, the results provide limited evidence that the simulator can reliably evaluate real-world counter-fraud strategies.

  • Limited experimental scale and robustness. The main experiments involve only 112 civilian agents and eight fraudster agents over 20 simulated days. Macro-level validation uses three random seeds, while the intervention comparisons use a single reference population and seed. Given the stochastic nature of LLM-based agents, this setup provides limited evidence of robustness across populations, random initializations, and simulation settings. Moreover, the oracle intervention relies on privileged information about executed payments and campaign states, so its strong performance should be interpreted as an idealized upper bound rather than evidence of a practically deployable intervention.

Questions for authors

Check the above weaknesses.

4 comments

W1: Division of Labor Between Civilian NIA and Multi-Stage Fraudster Modeling

Thank you for asking what specific role the Need-Intent-Action architecture plays and why it is necessary for financial-fraud simulation. Below, we clarify that NIA is used only for civilians while fraudsters follow a dedicated multi-stage scam strategy, explain why modeling a victim's everyday economic life is essential to simulating fraud, and show how removing NIA affects empirical alignment.

1. Civilians and Fraudsters Use Different Architectures for Different Roles.

We do not apply a single generic agent model to everyone. Instead, civilians and fraudsters use asymmetrical architectures tailored to their distinct roles:

  • Fraudsters: Fraudsters do not use NIA. Instead, they follow a hierarchical multi-stage fraud policy operating at two timescales. Each day, they plan how a scam campaign progresses across target recruitment, credibility building, peak money extraction, and eventual collapse. Each hour, they publish lures, steer private persuasion chats, and adjust small bait returns, payment demands, and withdrawal blocks based on what each victim has paid so far.
  • Civilians: Only civilian agents use the Need-Intent-Action architecture. Each hour, a civilian updates their psychological needs from recent events and account balance changes, selects an intent route driven by those needs, and then chooses a concrete action. This embeds scam exposure and payment decisions directly inside everyday routines of working, resting, spending, and socializing under a shared time budget and bank balance.

Thus, NIA is not a generic wrapper applied to every agent: fraudsters are governed by a dedicated multi-stage scam policy, whereas NIA is designed specifically to model how everyday civilian life interacts with fraud.

2. Why Financial-Fraud Simulation Specifically Needs NIA for Civilians.

In the real world, victims do not sit in a dedicated scam chatroom waiting to be defrauded—scams unfold in the middle of everyday financial pressures and social needs. NIA is necessary because it connects a civilian's normal daily activities, account balance, and evolving psychological needs with their exposure and response to scams:

  • Daily work and consumption shape financial vulnerability: Ordinary spending lowers a civilian's available balance and raises financial safety needs. When safety need is high, a civilian's transfer rate more than doubles from 0.94 to 1.94 per 100h; when social need is high, it rises from 1.03 to 1.91 per 100h.
  • Financial feedback loops back into daily needs: When a scammer pays a small bait return or blocks a withdrawal, the resulting balance change updates the civilian's needs in the next hour, shaping whether they seek verification, send additional money to recover sunk costs, or walk away.

NIA is essential to FraudSociety because it embeds scam exposure and payment decisions inside a civilian's everyday economic life, allowing financial stress, unmet social needs, and ledger feedback to dynamically shape vulnerability over time.

3. Ablation Evidence: Removing NIA Breaks Macro Fraud Alignment.

When we keep the exact same population, LLM action policy, need updates, and financial ledger feedback, and replace NIA's need-driven intent routing with uniform route sampling, the simulated fraud distribution breaks down:

Model VariantHTL↓ALS↓FSH↓ALR↓Composite QD↓Relative QD Change vs. w/o NIA
w/o NIA1.00000.65380.79080.18830.6582+0.0% Ablation Baseline
w/o Ledger Feedback0.70050.57260.34100.18980.4510−31.5% vs. w/o NIA
Full Model with NIA and Ledger0.08930.65130.30120.14950.2978−54.8% vs. w/o NIA

Our ablation confirms that without NIA linking daily needs to social activity and payment choices, the simulator cannot reproduce real-world heavy-tailed loss concentration or differences in severity across scam families.

W2: Validating Underlying Victimization Mechanisms and Real-World Counter-Fraud Strategies

Thank you for noting that matching aggregate Quartet Distance statistics alone does not validate underlying victimization processes or prove that simulated loss changes reflect real-world counter-fraud strategies. Below, we validate underlying mechanisms via a blind audit of all 81 victim pathways, report a 10-expert blind evaluation of 250 pre/post-warning trajectory pairs across C1–C5 alongside 900 controlled trials, and test a realistic non-oracle detector across 10 runs.

1. Validating Underlying Victimization Mechanisms Beyond Aggregate QD Statistics.

Three raters blindly reviewed all 81 positive-net-loss victim pathways, inspecting posts, private chats, payments, bait returns, and withdrawal requests. By majority vote of at least 2 of 3 raters, 68 of 81 pathways were judged realistic, while the 13 unapproved cases showed rapid trust progression without verification or repetitive payment excuses:

Fraud FamilyRecordsVictimsLoss ShareLagApproved ≥2/3Split 3/2/1/0
Investment Fraud191861.6%, $89.4K114h16/19, 84.2%14/2/1/2
Impersonation Fraud23208.0%, $11.6K50h20/23, 87.0%17/3/0/3
Marketplace Fraud343127.9%, $40.5K52h28/34, 82.4%24/4/1/5
Romance Fraud552.5%, $3.6K203h4/5, 80.0%3/1/0/1
All Pathways Combined8163100.0%, $145.0K72h68/81, 84.0%58/10/2/11, κ=0.68

Preserving every per-transfer amount Pv,c,tP_{v,c,t} across 10 runs while shuffling transfers across contacted civilians worsens QD from 0.3238±0.0395 to 0.4755±0.0712 and HTL from 0.1120 to 0.5260.

Blind pathway evaluation and transfer shuffling confirm that our macro patterns arise from coherent multi-step persuasion and financial feedback on specific victims.

2. Ten-Expert Blind Evaluation of Counter-Fraud Strategies and Micro Warning Responses.

To test whether agent responses to warnings reflect realistic human behavior, 10 anti-fraud practitioners blindly classified 250 pre/post-warning trajectory pairs—50 randomly sampled warnings per strategy across C1–C5—as Real or Fake, totaling 2,500 judgments. Overall, 2,029 of 2,500 ratings, or 81.2%, judged the trajectories Real vs. 471 Fake, with individual experts ranging consistently from 78.0% at 195/250 to 84.8% at 212/250. By majority vote, 215/250 or 86.0% of pairs were judged Real, with 33 Fake and 2 5:5 ties in C3 and C4:

Strategy, 50 Pairs EachRealFakeReal ShareMajority Approved ≥6/10
C1: Platform Broadcast4217984.2%46/50, 92.0%
C2: Monitor Labels4168483.2%44/50, 88.0%
C3: Random Comments39011078.0%41/50, 82.0%
C4: Peer Warnings4118982.2%44/50, 88.0%
C5: Oracle Personalized39110978.2%40/50, 80.0%
Overall, 2,500 Judgments2,02947181.2%215/250, 86.0%

Blind evaluation by 10 anti-fraud practitioners across all five strategies confirms that simulated warning responses closely match real-world counter-fraud behavior.

3. Multi-Run Evaluation of a Realistic, Non-Oracle Counter-Fraud Pipeline.

Across 10 paired runs of 1,120 civilians, we tested a Detector-Driven Warning Policy using only visible messages and the user's payment history—at 73.9% Precision, 68.0% Recall, 6.0% FPR, and 2h delay, reducing legitimate payments by 2.1%:

Condition, 10 Runs of 1,120 CiviliansLoss Red.95% CIRepeat CiviliansRepeat Share
C0: No Warning Baseline0.0% Ref.0.0% to 0.0%346/1,12030.9%
C1: Platform Broadcast30.7%22.8% to 38.6%243/1,12021.7%
C2: Monitor Labels39.9%33.3% to 46.5%272/1,12024.3%
C3: Random Comments64.0%57.6% to 70.4%196/1,12017.5%
C4: Peer Warnings76.2%71.6% to 80.8%136/1,12012.1%
Detector-Driven Realistic59.8%54.3% to 65.3%194/1,12017.3%, Δ −13.6 pp
C5: Oracle Personalized82.6%79.0% to 86.2%78/1,1207.0%

Combining pathway audits, 10-expert blind evaluation of warning responses, and 10-run detector testing shows that FraudSociety reliably evaluates real-world counter-fraud strategies.

W3: Multi-Run Robustness Across Population Scale, Settings, and Information Access

Thank you for emphasizing the need to test larger populations, run multiple random seeds for both macro validation and interventions, and evaluate realistic warnings that do not rely on oracle ground truth. Below, we expand our evaluation to 10 matched runs per condition across a tenfold population scale-up, different network topologies, longer durations, and alternative LLM backbones, and compare a realistic detector-driven intervention against the C5 oracle upper bound.

1. Multi-Run Robustness Across 10× Population Scale, Topologies, Horizons, and LLM Backbones.

To ensure our findings are not an artifact of a small 112-civilian population or a few random seeds, we ran 10 matched initial populations and network realizations per setting. We tested a tenfold scale-up to 1,120 civilians and 80 fraudsters, degree-matched random and community networks, extended 40-day and 60-day horizons, census age-stratified sampling, and two additional LLM backbones, Qwen3-32B and Llama-3.3-70B-Instruct:

Robustness Setting across 10 RunsHTL↓ALS↓FSH↓ALR↓Mean QD±SD↓C4 Net-Loss Red.C5 Net-Loss Red.Detector-Driven Loss Red.
Baseline 112+8 over 20d on Qwen3-14B0.11200.66200.34700.17420.3238±0.039576.2%82.6%59.8%
Tenfold Scale 1,120+80 over 20d0.11500.66500.35400.17120.3263±0.025574.6%80.8%58.4%
Degree-Matched Random Network0.13800.67800.38600.18800.3475±0.039073.1%80.2%57.2%
Community Network0.15600.69200.40800.19640.3631±0.042571.8%79.1%55.9%
40-Day Horizon with Fixed Campaigns0.14200.68500.39500.18960.3529±0.040672.9%79.4%56.5%
60-Day Horizon with Fixed Campaigns0.17400.71200.42800.20400.3795±0.046970.5%77.2%54.1%
ACS Age-Stratified Population0.12800.65100.34200.16060.3204±0.024874.8%81.1%58.9%
Backbone: Qwen3-32B0.10400.63800.32900.16420.3088±0.033877.8%83.5%61.5%
Backbone: Llama-3.3-70B-Instruct0.13600.68400.38200.18760.3474±0.042670.8%78.7%56.2%

Expanding the simulation tenfold and testing across multiple random seeds, network structures, time horizons, and LLM backbones confirms that both our macro fidelity and the relative ordering of warning interventions remain stable and are not artifacts of a small population or a single seed.

2. 10-Seed Intervention Confidence Intervals and Realistic Detector vs. Oracle Upper Bound.

We evaluated all warning strategies across 10 paired runs covering 1,120 initial civilians, where C5 beats C4 in 8 of 10 runs and C1 and C2 swap ranks in 3 of 10 runs. To address the concern that C5 uses privileged oracle information, we explicitly treat C5 as a theoretical upper bound and compare it against a Detector-Driven Warning Policy that observes only visible messages and the user's own transaction history—operating at 73.9% Precision, 68.0% Recall, a 6.0% false-positive rate, and a 2h median trigger delay:

Condition across 10 Paired Runs of 1,120 CiviliansInformation AccessMean Loss Red.95% CICohort Repeat CiviliansFixed-Cohort Repeat Share
C0: No WarningNone, Baseline0.0% Ref.0.0% to 0.0%346/1,12030.9%
C1: Platform BroadcastDaily Generic Broadcast30.7%22.8% to 38.6%243/1,12021.7%
C2: Monitor LabelsPublic Post Screening39.9%33.3% to 46.5%272/1,12024.3%
C3: Random Comments5% Random Post Comments64.0%57.6% to 70.4%196/1,12017.5%
C4: Peer WarningsBounded Civilian View76.2%71.6% to 80.8%136/1,12012.1%
Detector-Driven WarningVisible Msgs and Own Ledger, Realistic59.8%54.3% to 65.3%194/1,12017.3%, Δ −13.6 pp
C5: Oracle PersonalizedSimulator Ground Truth, Upper Bound82.6%79.0% to 86.2%78/1,1207.0%

Evaluating all interventions across 10 paired seeds and introducing a realistic detector-driven policy clearly separates what a deployable warning system can achieve under imperfect detection from the theoretical upper bound of an oracle intervention.

We hope the above clarifications and the additional experiments sufficiently addressed your concerns. If you are satisfied, we kindly request you to consider updating the score to reflect the newly added results and discussion. We remain committed to addressing any remaining points you may have during the discussion phase.

Reviewer_9VTVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Fraud Society, an LLM-based simulator that couples social interactions with persistent financial states to model financial fraud. Civilian agents follow a Need–Intent–Action framework, while fraudsters adapt their communication and financial actions to interaction histories. The paper evaluates aggregate loss statistics, individual victimization pathways, and warning interventions.

Critical strengths
  1. Interesting and practically relevant setting. Modeling how persuasion, payments, and subsequent financial feedback interact is a meaningful research direction. Following victimization beyond the first payment is particularly useful for studying repeated losses and opportunities for intervention.

  2. Clear and reasonable framework. The separation between civilians, fraudsters, the social environment, and financial ledgers is easy to follow. Persistent records of payments, returns, and withdrawal requests provide a coherent way to connect earlier experiences with later decisions.

  3. The paper examines aggregate statistics, individual trajectories, and mechanism ablations. The recorded interaction and transaction histories also make simulation outcomes inspectable, which is valuable for understanding where the observed patterns come from.

Critical weaknesses
  1. The experimental scale provides limited support for population-level conclusions. Each simulation contains only 112 civilians and eight fraudsters over 20 days. A small simulation can still be informative, but the paper does not establish whether its conclusions remain stable as population size, network structure, or duration changes. Some analyses rely on particularly small samples: the reference run contains only five positive-loss romance-fraud records. Population representativeness is also uncertain. In Table 5, civilians aged 50–59 constitute 28.6% of the reference population, compared with the 16.5% sampling target. Three regenerated populations provide some robustness evidence, but the interventions are evaluated on only one population and seed.

  2. Behavioral generalization remains insufficiently demonstrated. The simulator covers four fraud families, with two campaigns per family, and imposes a shared progression through recruiting, credibility, peak extraction, and collapse. This is a reasonable starting point, but it remains unclear how well the framework accommodates different campaign structures, unfamiliar fraud strategies, or longer victimization processes. Civilian profiles include demographic and financial attributes in addition to Big-Five scores; however, assigning these attributes does not itself establish realistic behavioral diversity. The paper provides limited independent evidence that the resulting agents reproduce human differences in trust, verification, payment decisions, and warning uptake. All reported experiments also use a single LLM backbone.

  3. Practical usefulness needs stronger demonstration. For fraud-detection research specifically, the paper does not demonstrate that simulated data improve detector training, support reliable algorithm comparisons, or produce findings that transfer to real data. An experiment connecting the simulator to an actual detection or intervention pipeline would substantially strengthen the contribution. The warning experiments do not yet show that the simulator has practical utility for fraud prevention; they mainly demonstrate that the simulator can generate internally consistent responses to hypothetical interventions. In particular, C5 assumes access to the true fraud status and campaign information after a payment. Although the paper appropriately identifies this as an oracle intervention, its effectiveness does not establish what a realistic system could achieve with imperfect detection. Intervention comparisons are also single-run results, and timing, delivery channel, and information specificity vary together. Thus, these experiments should be interpreted as proof-of-concept stress tests of the simulated population rather than evidence that the simulator itself can support deployed warning systems.

  4. The evidence for high fidelity and superiority over existing simulators needs clearer qualification. Appendix D describes age- and fraud-family-specific amount guidance, while Appendix L states that regulator-facing diagnostics informed iterative development. Consequently, agreement with related aggregate statistics may partly reflect the construction process. The paper should distinguish calibration targets from independent validation evidence; matching several aggregate patterns alone does not establish that the underlying behavioral mechanisms are realistic. The baseline comparison also has limitations: baseline results come from single runs, the systems use different loss-accounting units, and TransXion’s QD averages three components rather than four. The component results are informative, but a common evaluation protocol would make the claimed advantage easier to assess.

Questions for authors
  1. Why were 112 civilians chosen, and what population are they intended to represent? How sensitive are the results to population size and composition, particularly given the differences between the target and realized age distributions in Table 5?

  2. How are civilian personas validated beyond assigning demographic, economic, and Big-Five attributes? Is there evidence that the personality scores produce realistic and stable differences in fraud susceptibility and warning response? How would factors such as financial literacy or previous fraud experience be incorporated?

  3. What concrete research task benefits most from Fraud Society compared with TransXion and MultiAgent4Fraud? Could the authors demonstrate this advantage through a shared downstream task, such as evaluating a fraud detector or a warning policy with realistic information access?

6 comments

W1 and Q1: Population Scale, Age Composition, and Multi-Seed Statistical Robustness

Thank you for highlighting population scale, age representativeness, and multi-seed stability. Below, we explain why we chose the 112-civilian baseline, examine how the simulator behaves when scaled tenfold to 1,120 civilians and 80 fraudsters across 10 runs, test whether age-distribution skew affects our results, and evaluate all interventions across 10 paired seeds.

1. Why 112 Civilians, Target Population, and Tenfold Scaling Across Graphs and Horizons.

Our civilians represent the U.S. adult population aged 20 and older, initialized from U.S. Census ACS age distributions and Federal Reserve SCF balances. Because each agent executes 480 hourly closed-loop LLM steps over 20 days, we used 112 civilians and 8 fraudsters—two per family—as a tractable baseline for extensive ablations. To verify stability across scale, network topology, and duration, we ran 10 independent runs per setting, including a tenfold expansion to 1,120 civilians and 80 fraudsters:

Setting across 10 RunsHTL↓ALS↓FSH↓ALR↓Mean QD±SD↓Top-10%LagC4 / C5 Red.
Baseline 112+8, 20d0.11200.66200.34700.17420.3238±0.039579.0%76h76.2% / 82.6%
Tenfold 1,120+80, 20d0.11500.66500.35400.17120.3263±0.025577.4%81h74.6% / 80.8%
Degree-Matched Graph0.13800.67800.38600.18800.3475±0.039076.5%79h73.1% / 80.2%
Community Graph0.15600.69200.40800.19640.3631±0.042575.8%84h71.8% / 79.1%
40-Day Fixed Campaigns0.14200.68500.39500.18960.3529±0.040680.2%78h72.9% / 79.4%
60-Day Fixed Campaigns0.17400.71200.42800.20400.3795±0.046981.5%80h70.5% / 77.2%

As the table shows, scaling tenfold keeps macro alignment and warning gains virtually unchanged, while denser graphs or longer horizons without new campaigns introduce only modest drift. Pooling the tenfold runs also provides a large romance-fraud cohort, confirming that long cultivation lags and frequent repeat payments hold at scale:

Romance-Fraud CohortCampaignsPos.-Loss RecordsMedian Lag, IQRRepeat ShareMedian Loss
Reference Run 112+825203h, IQR 122–215h20.0%, 1/5$600
Tenfold Pooled, 10 Runs200387218h, IQR 151–307h35.4%, 137/387$650

In direct response to your concern about scale, expanding the population tenfold and varying network topologies or horizons leaves both macro loss alignment and warning effectiveness stable, while confirming that our romance-fraud patterns are genuine structural behaviors rather than small-sample noise.

2. Correcting the Age Distribution via Stratified Sampling.

In Table 5, the reference population over-sampled ages 50–59 due to finite-sample variance in one unstratified draw. To test whether this skew drives our results, we compared the reference run and 10 unstratified draws against 10 age-stratified runs matching U.S. Census ACS brackets:

Population Cohort50–59 ShareHTL↓ALS↓FSH↓ALR↓QD↓C4 / C5 Red.
Table 5 Reference, 1 Run28.6%, 32/1120.08930.65130.30120.14950.297882.9% / 87.6%
Unstratified, 10 Runs17.9%±4.2%0.11200.66200.34700.17420.3238±0.039576.2% / 82.6%
ACS Stratified, 10 Runs16.1%, 18/1120.12800.65100.34200.16060.3204±0.024874.8% / 81.1%

Correcting the 50–59 age over-representation via census-stratified sampling slightly improves age alignment while leaving macro fraud patterns and intervention conclusions intact, showing that our findings are not driven by demographic sampling skew.

3. Multi-Seed Intervention Stability Across 10 Paired Runs.

We expanded all warnings—plus a realistic Detector-Driven Warning Policy without oracle access—to 10 paired runs covering 1,120 civilians, where C5 beats C4 in 8 of 10 runs and C1 and C2 swap ranks in 3 of 10 runs:

Condition, 10 Runs of 1,120 CiviliansLoss Red. vs. C095% CIRepeat CiviliansCohort Repeat Share
C0: No Warning Baseline0.0% Ref.0.0% to 0.0%346/1,12030.9%
C1: Platform Broadcast30.7%22.8% to 38.6%243/1,12021.7%
C2: Monitor Labels39.9%33.3% to 46.5%272/1,12024.3%
C3: Random Comments64.0%57.6% to 70.4%196/1,12017.5%
C4: Peer Warnings76.2%71.6% to 80.8%136/1,12012.1%
Detector-Driven Realistic59.8%54.3% to 65.3%194/1,12017.3%, Δ −13.6 pp
C5: Oracle Personalized82.6%79.0% to 86.2%78/1,1207.0%

Across 10 paired random seeds, confidence intervals remain tight and the ranking of warning interventions stays consistent, confirming that our intervention conclusions are not artifacts of a single run.

W2 and Q2: Behavioral Generalization, Persona Validation, and Cross-Model Stability

Thank you for asking whether our simulator can accommodate unfamiliar scam structures, generalize across LLM backbones, and produce realistic human behavioral differences. Below, we test the frozen simulator on eight unseen scam structures, replicate our experiments across three LLM backbones, and validate individual agent behavior through blind human review of all 81 victim pathways alongside 900 controlled persona trials.

1. Generalizing to Unseen Scam Structures Beyond a Fixed Four-Stage Template.

The four campaign stages in our paper describe high-level fraud phases rather than a rigid script. Holding the simulator code and agent prompts unchanged, we replaced the default campaigns with 8 unseen campaign designs across the four fraud families—including single-contact immediate-payment lures that skip relationship building, no-trial-return schemes that omit bait payouts, and long-horizon relationship grooming:

Unseen Campaign Archetype across 10 RunsInstantiatedInteractivePaid CampaignsMedian Contact-to-First-Pay Lag
Single-Contact Immediate Payment2015, 75.0%11, 55.0%8h
No-Trial-Return Schemes3026, 86.7%19, 63.3%57h
Long-Horizon Relationship Grooming3028, 93.3%20, 66.7%221h
All Unseen Configurations Combined8069, 86.3%50, 62.5%64h Pooled Median

Without any code changes or retuning, the simulator naturally adapts to unfamiliar scam structures—capturing both rapid conversions in immediate-payment lures and extended cultivation in long-horizon grooming, while allowing mismatched scam attempts to fail naturally when civilians ignore or reject them.

2. Stability Across Alternative LLM Backbones.

To verify that our findings are not tied to a single model, we ran the same evaluation using Qwen3-32B and Llama-3.3-70B-Instruct:

LLM Backbone across 10 RunsHTL↓ALS↓FSH↓ALR↓Mean QD±SD↓Top-10% LossC4 Loss Red.C5 Loss Red.
Qwen3-14B Main Backbone0.11200.66200.34700.17420.3238±0.039579.0%76.2%82.6%
Qwen3-32B0.10400.63800.32900.16420.3088±0.033880.4%77.8%83.5%
Llama-3.3-70B-Instruct0.13600.68400.38200.18760.3474±0.042676.8%70.8%78.7%

Both macro loss alignment and warning intervention rankings hold consistently across different model families and parameter scales, demonstrating that the simulated fraud dynamics come from our interaction framework rather than a specific LLM backbone.

3. Validating Individual Victim Trajectories and Persona Behavioral Diversity.

To validate personas beyond assigning profile labels, three raters blindly reviewed all 81 positive-net-loss victim trajectories—inspecting posts, private chats, payments, bait returns, and withdrawal attempts. Raters judged the vast majority of trajectories across every scam family to be realistic, while the 13 unapproved cases mainly showed rapid trust progression or repetitive payment excuses:

Fraud FamilyRecordsVictimsApproved ≥2/3RateSplit 3/2/1/0
Investment Fraud19181684.2%14/2/1/2
Impersonation Fraud23202087.0%17/3/0/3
Marketplace Fraud34312882.4%24/4/1/5
Romance Fraud55480.0%3/1/0/1
All Pathways Combined81636884.0%58/10/2/11, κ=0.68

To test whether personality scores drive stable, realistic differences and how Financial Literacy and Prior Fraud Experience can be incorporated, we added both attributes to the civilian persona schema and NIA prompt. Across 900 controlled trials per condition spanning 50 scenarios, 3 prompt rephrasings, and 6 repeats, civilians maintain 79.3% agreement across rephrasings at 714/900 and respond systematically when individual traits are varied:

Profile Manipulation across 900 DecisionsBehaviorBaseline/LowHigh/TreatedEffect Δ pp95% CI
Conscientiousness High vs. LowVerification394/900, 43.8%469/900, 52.1%+8.3 pp+3.2 to +13.5 pp
Financial Literacy Lvl 4–5 vs. 1–2Verification382/900, 42.4%477/900, 53.0%+10.6 pp+4.0 to +17.1 pp
Prior Fraud Experience Yes vs. NoWarning Uptake512/900, 56.9%568/900, 63.1%+6.2 pp+0.8 to +11.6 pp
Neuroticism High vs. LowPayment Rate216/900, 24.0%229/900, 25.4%+1.4 pp−3.8 to +6.6 pp

Together, blind human review of individual victim pathways and controlled persona experiments confirm that our civilian agents do not merely carry demographic labels—they make stable decisions across prompt rephrasings and reproduce realistic human differences in verification and warning uptake as conscientiousness, financial literacy, and prior fraud experience change.

W3 and Q3: Practical Utility for Pre-Deployment Policy Simulation and Detector-Driven Interventions

Thank you for asking how FraudSociety provides practical value for fraud prevention and what concrete task benefits most compared with existing simulators. Below, we show how FraudSociety serves as an interactive pre-deployment sandbox to test detection-and-warning policies before costly real-world rollout, connect a realistic non-oracle detector across 10 runs, and isolate warning content, timing, and delivery channel.

1. Practical Utility: Pre-Deployment Simulation Before Costly Real-World Rollout.

Offline datasets can train static classifiers, but cannot show what happens after an intervention is triggered—whether victims heed warnings, whether scammers pivot to new excuses for follow-up payments, or how false alarms disrupt normal users. Meanwhile, A/B testing unproven warnings on live platforms is costly and exposes real users to financial harm. FraudSociety enables pre-deployment counterfactual evaluation of anti-fraud policies, especially for interrupting post-payment escalation after a first transfer—a task existing simulators cannot support:

  • TransXion: Simulates transaction graphs without social feeds, chats, or warnings, so it cannot test communication-based interventions.
  • MultiAgent4Fraud: Simulates scam chats and warnings, but terminates at the first transfer, making it impossible to evaluate repeat payments, bait returns, or post-payment intervention:

Rather than serving only as a static data generator, FraudSociety provides a pre-deployment testbed where practitioners can simulate closed-loop scam escalation and evaluate how candidate interventions curb repeat victimization before costly real-world rollout.

2. Pre-Evaluating a Realistic, Imperfect Fraud Detector Against the C5 Oracle Upper Bound.

While C5 represents an idealized oracle upper bound, FraudSociety can directly evaluate deployable detection pipelines with imperfect accuracy. We connected a Detector-Driven Warning Policy observing only visible messages, public risk cues, and the user's own payment history. Across 10 runs, it operates at 73.9% Precision, 68.0% Recall, 6.0% FPR, and a 2h median delay, issuing 16.4 alerts per 100 civilians and reducing legitimate payments by 2.1%:

Condition across 10 Runs of 1,120 CiviliansMean Loss Red.95% CICohort Repeat CiviliansFixed-Cohort Repeat Share
C0: No Warning Baseline0.0% Ref.0.0% to 0.0%346/1,12030.9%
C1: Platform Broadcast30.7%22.8% to 38.6%243/1,12021.7%
C2: Monitor Labels39.9%33.3% to 46.5%272/1,12024.3%
C3: Random Comments64.0%57.6% to 70.4%196/1,12017.5%
C4: Peer Warnings76.2%71.6% to 80.8%136/1,12012.1%
Detector-Driven Realistic59.8%54.3% to 65.3%194/1,12017.3%, Δ −13.6 pp
C5: Oracle Personalized82.6%79.0% to 86.2%78/1,1207.0%

Connecting an imperfect detector shows how practitioners can pre-evaluate the trade-off between fraud loss reduction, repeat victimization, and false-positive friction on normal payments, while quantifying the exact headroom relative to the C5 oracle upper bound.

3. De-Confounding Content Specificity, Timing, and Delivery Channel.

To show how the simulator screens warning designs before deployment, we fixed detector triggers across 10 paired runs and varied only one factor at a time:

Isolated Factor across 10 RunsBaseline LevelTarget LevelMarginal Gain95% CICohort Repeat Shift
Content: Contextual vs. GenericGeneric 47.0%Contextual 59.8%+12.8 pp+8.4 to +17.2 pp23.4% → 17.3%, −6.1 pp
Timing: Early vs. DelayedDelayed 52.8%Early 59.8%+7.0 pp+4.3 to +9.7 pp20.8% → 17.3%, −3.5 pp
Channel: Private DM vs. PublicPublic 56.6%Private DM 59.8%+3.2 pp−1.8 to +8.2 pp18.6% → 17.3%, −1.3 pp

Isolating individual warning dimensions resolves the confounding in C1–C5 and provides a clear pre-deployment design guideline: investing in context-specific warning content and early trigger timing yields far greater fraud reduction than changing the delivery channel alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.