acceptodds
Under review as a conference paper at ICLR 2027

STAR-Bench: Evaluating Anti-Money Laundering Agents for Regulatory Reporting Workflows

Abstract

Anti-money laundering (AML) investigations are regulated tool-using workflows that end in a suspicious transaction report (STR), and they are harder than general tool use in four ways: tools reading the same fields serve different regulatory roles, reporting is an obligation, evidence must carry across turns, and models are chosen by proxies whose bearing on this work is unknown. Existing function-calling and financial benchmarks do not score whether an agent carries out a mandated reporting step. We introduce STAR-Bench, a compliance-oriented benchmark derived from an AML agent platform and expert-verified scenarios: 23 AML tools and 1,308 evaluation cases spanning single-turn tool use and multi-turn STR workflows, where every tool executes against the database. Experiments on 28 open-weight configurations show that single-turn tool accuracy does not resolve workflow readiness: configurations within seven points on single-turn tool hit differ by thirty-two points in STR workflow completion, and the best completer ranks 17th of 28 on single-turn hit. Workflow failures concentrate at the STR-writing turn and persist under an oracle that injects executed tool outputs, while end-to-end execution costs parameter grounding rather than tool selection. Reporting tools are about as hard to trigger as analysis tools on average but fail differently: three quarters of STR-field validation failures are calls never made. General function-calling rank, finance specialization and schema language give no reliable ordering of these results, and reasoning mode lowers them on average under our output budget. We release the STAR-Bench dataset, executable evaluation environment, evaluation code, and AML agent platform.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agents, tool use, function calling, multi-turn evaluation, anti-money laundering, financial compliance, regulatory reporting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_B6bjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces STAR-Bench, a compliance-oriented benchmark for evaluating LLM agents on anti-money laundering (AML) regulatory reporting workflows. The authors argue that AML investigations differ from general tool use along four axes: overlapping tool roles (C1), a mandated reporting step (the suspicious transaction report, STR) that must be scored as an action rather than prose (C2), evidence that must carry across turns (C3), and model selection driven by proxies (function-calling leaderboards, specialization labels, schema language, reasoning mode) whose bearing on this workflow is unknown (C4).

Critical strengths
  1. A benchmark design that measures a mandated reporting step as an action, with execution-grounded evaluation throughout. Scoring the STR step as a tool call (C2), grading the produced report against the outputs of the turns before it, executing every tool against a real database, and injecting executed gold outputs in the oracle setting together produce an evaluation in which failure attribution is clean: the paper can show the workflow breaks specifically at the report-writing turn (0.564 vs 0.859 hit at turn 4), surviving an oracle that erases all earlier mistakes. This is consequential because it converts a vague practical worry ("agents can't do compliance") into a located, reproducible measurement — and none of BFCL, ToolSandbox, τ-bench, or the financial benchmarks score anything like it (Table 1).

  2. A rigorously supported negative result with direct deployment relevance: proxies do not order agents on this workflow. The claim that single-turn hit does not resolve workflow readiness is backed by a concrete, statistically tested spread (Δc = 0.32 within a 7-point single-turn band; paired bootstrap [16,44] and [20,52] with McNemar p ≤ 0.0007), and the same discipline is applied to BFCL rank correlation, specialization pairs, the 2×2 language ablation, and thinking/non-thinking pairs. This matters because model selection for regulated on-premises deployment is done exactly by these proxies; showing their unreliability on a benchmark built from the actual regulated workflow is the kind of evidence that changes procurement and evaluation practice, and it makes the benchmark useful even to groups that never touch AML.

Critical weaknesses
  1. All workflow-level conclusions rest on 50 scenarios (only 45 ending in an STR turn; 22 long-context, 14 base, 14 missing-parameter). Completion is a binary per scenario, so per-configuration c estimates have ±10+ point uncertainty, and the finding that "the best completer ranks 17th on single-turn hit" — a centerpiece of Figure 1 and the abstract — depends on the particular scenario mix. The authors test differences between selected configuration pairs, but no scenario-count or scenario-composition sensitivity analysis is reported (e.g., does the Δc = 0.32 spread survive leave-one-fraud-type-out?). This is consequential because the paper's main claim is precisely that workflow-level evaluation reveals what single-turn evaluation misses; the claim is only as sturdy as the 50 scenarios.
  2. The STR checker is keyword/regex-based; its "grounding" dimension checks that a figure appeared in some earlier tool output without verifying the figure is bound to the correct account/transaction, and its validation (Appendix F) uses two raters who are authors, on 45 reports, with a floor effect on the terminology dimension (much of the evidence-match agreement comes from the shared zero convention — dropping no-figure reports leaves ρ ≈ 0.33–0.42). Since "writing an STR grounded in the evidence" is one of the benchmark's headline capabilities (C3/C2), the community will cite these checker scores; the current validation is not strong enough to support that weight, and an independent (non-author) rater study or an LLM-as-judge cross-check would materially change confidence in Table 3.
Questions for authors
  1. In the executed-oracle setting, after a turn where the model deviates from the gold call, the next turn receives the gold call's executed output. How do you handle cases where the model's earlier deviation makes the gold next call semantically incoherent for the model (e.g., it never retrieved the account the gold Turn-3 call references)? Are such scenarios excluded, scored as failures, or does the injected context simply "reset" the investigation? This bears directly on how the turn-4 drop should be interpreted.

  2. Can you report a sensitivity analysis of the headline workflow claims to scenario composition — e.g., the Δc spread and the best-completer ranking under leave-one-scenario-type-out or bootstrap over the 50 scenarios? With c binary over n = 50, what is the confidence interval on Mistral-Small-24B's c = 0.72 itself?

  3. The reasoning-mode conclusion (Section 5.3) rests on four thinking/non-thinking pairs and an explicit output budget, with budget exhaustion as the stated mechanism. What was the budget, and does the −2.8 / −7.5 effect persist at a larger budget? If the effect is budget-bound, the claim should be scoped accordingly in the abstract, where "reasoning mode lowers them on average" currently reads as budget-free.

  4. For missing-parameter multi-turn scenarios, how exactly is a "correct clarification" scored — is any no-call response credited, or must the model ask for the specific missing arguments? A generous rule here would inflate turn-level hit.

7 comments

Response to Reviewer B6bj (1/4)

Thank you for the detailed questions on scenario composition, evidence attribution and reasoning mode. They led us to add sensitivity analyses and to state the scope of several claims more precisely. The revised PDF marks all revisions in blue. This comment answers W1 and Q2, the second answers W2, the third answers Q1, and the fourth answers Q3 and Q4.

(W1 & Q2) Sensitivity to scenario composition, and the interval on c = 0.72

Yes. The completion spread survives every tested scenario-type and fraud-type removal, while the leading configuration changes with the scenario type. The data thus support a difference between configurations but not a single best one. Mistral-Small-24B's c is 36 of 50, with a nominal Wilson 95% interval of 58.3% to 82.5%.

Leaving out one fraud type at a time, as you suggest, keeps the spread between 30.0 and 41.7 points. Mistral-Small-24B stays first in all six cases, tied in two. The table removes one scenario type at a time. The spread is over the fixed band of ten configurations with similar single-turn hit, and the rank over all 28.

ScenariosCountSpread of c (points)Rank of Mistral-Small-24B among 28
All5032.01
Without base3638.91
Without missing-parameter3636.13
Without long-context2846.42

Paired comparisons put the spread beyond scenario sampling alone. The within-scenario permutation test gives p = 0.006 for the 32-point spread, with a null 95th percentile of 26 points. It keeps the ten configurations paired on the same scenarios, and each scenario-type removal gives p below 0.02. Your concern about uncertainty above ten points holds for one configuration's c, but the paired intervals you cite, [16, 44] and [20, 52] points, exclude zero.

Two band configurations share the lowest c. One of them, Qwen3.5-27B (T), completes 48% when rerun in a separate collection, so Section 4.4 now compares the leaders with the other, Qwen3.6-35B-A3B. The paired intervals become [18, 42] and [20, 52].

The identity of the winner also varies under resampling. The Wilson interval is not corrected for selecting the winner. In 10,000 paired resamples of the 50 scenarios over the 28 configurations, Mistral-Small-24B is uniquely first in 44.5% and first including ties in 56.2%. Both are resampling frequencies rather than probabilities that it is better in the population.

We removed the rank of the best completer from the Abstract, and Figure 1, Section 4.4 and the Conclusion now state that the leading configuration changes with the scenario mix. The strict criterion requires the applicable arguments and no tool outside the reference at every turn. Its spread in the band also exceeds scenario sampling (p = 0.001). Within the band of ten configurations with similar single-turn hit, Spearman's rho between single-turn hit and completion is 0.79 under the strict criterion and −0.23 under c. Among these configurations, single-turn hit tends to predict completion with correct arguments but not tool-selection completion, and Section 4.4 now makes only this narrower claim. Tool-selection completion asks whether the agent takes every step the workflow requires, including the report. Correct arguments at other turns do not make up for a skipped step.

Response to Reviewer B6bj (2/4)

(W2) Evidence attribution and checker validation

You are right that numerical overlap cannot show that a figure belongs to the correct account or transaction. We ran both checks you suggest, ratings by practitioners outside the author team and an LLM cross-check, and added a row-level check and the controlled tests below. The two author raters give overall correlations of 0.865 and 0.827, but as you noted, their evidence-match correlations fall to 0.417 and 0.326 on the 32 summaries with extracted figures, and their terminology scores sit near the floor.

Three AML practitioners outside the author team, with 4, 5 and 3 years at financial institutions, rated the same 45 reports independently, without model names. They read the whole draft, its transaction list included, while the checker reads only the summary. Against the mean of their ratings, Spearman's rho is 0.66 for required fields, 0.58 overall, 0.32 for terminology and 0.24 for grounding, the last not significant. Each practitioner, set against the mean of the other two, exceeds the checker by 0.03 on average for required fields and 0.09 overall, but by 0.24 for grounding and 0.27 for terminology. All five raters order the three configurations on the overall score as the checker does. A practitioner flags misattribution in 5 of the 45 reports, and two agree on only one. At least two flag an important omission in 20, including 7 of the 18 with checker grounding 1. Appendix F now reports these ratings beside the authors'.

In the row-level check, a figure written in the same clause as an account or transaction identifier is correct when one preceding tool-output row holds both. Only 241 of 758 Oracle reports contain such a pair. Of their 595 checkable figures, 3.9% are paired with a different row, and a hand check confirms five of ten such flags. Another 12.6% are absent from the tool outputs, a distinct error category. Only 9 of the 45 rated reports are checkable. The check finds no wrong pairing in them while the practitioners flag three, so we report it as a sensitivity rather than a validated score.

To test the checks themselves, we built 632 controlled reports, each paired with its evidence and with the required transactions labeled by SQL. The reports are generated from fixed English and Korean grammars. Altered reports change the account, amount, direction or date of a transaction, omit a required transaction or add one. The table counts the reports each check would accept among the 504 held out, which use 16 factual goals kept out of development.

Required factsCasesFull numerical overlapRow check, no flagsTransaction and coverage pass
Supported and complete66666666
Misbound, missing or extra rows438438960

Every altered report keeps full numerical overlap, and 246 contain exactly the same numbers as the correct report. The row check still accepts 96, and only checking each full transaction and the coverage of required rows rejects all 438. Independent Python and SQL labels agree on all 632 reports. The controls show that these constructed defects can be detected on new transactions in two grammars, and the follow-up appendix now includes them.

Two LLM judges assessed 156 proposed summaries, including the original 45, and each flag cites the IDs of the report and evidence spans it rests on. The protocol checks the format and consistency of a judgment but not whether it is correct. Both judges passed 64 synthetic controls and eight ID-sensitivity pairs.

Of 156 summariesGemini 2.5 ProClaude Haiku 5.5
Protocol-valid judgments145137
Flagged a potential problem90129
Flagged, checker grounding 141 of 63 (65%)55 of 59 (93%)
Flagged, checker grounding below 149 of 82 (60%)74 of 78 (95%)

The judges flag summaries that the checker scores 1 about as often as the others, and they agree on 84 of the 127 summaries that both judged validly. The cross-check therefore does not show that the grounding score measures attribution.

In the main text the checker scores appear only in Table 3 and its paragraph in Section 4.3, and neither the Abstract nor the Conclusion uses them. C3 is tested through the arguments of the multi-turn scenarios, and the reporting step of C2 is scored as a tool call, as your first strength notes.

In Section 4.3, Table 3 and the follow-up appendix we keep the original-extractor scores, recomputed after the loader correction. Section 3.5 now states that the checker reads only the summary. We report grounding with 16-digit account numbers counted, the row check and the LLM flags as separate sensitivities, and state that transaction-linkage accuracy is not validated. The reporting-turn drop in Section 4.4 and the no-call, fallback and keyword failures in Section 4.3 come from tool calls and do not depend on the checker.

Response to Reviewer B6bj (3/4)

(Q1) What the Oracle setting does after a deviation

Oracle resets the history to the reference calls and outputs after every turn, including after a deviation. No scenario is excluded. The model's deviating call is not executed or carried into the next turn, and it is scored as a miss at that turn. Thus the account referenced by a gold Turn-3 call appears in the history through executed reference outputs, even if the model never retrieved it. Oracle measures next-action selection on the reference path, while end-to-end (E2E) runs test the consequences of the model's own decisions. Section 4.4 now states this handling.

The reporting-turn drop persists with reference history at every turn, so it is not accumulated from earlier deviations. At the fourth turn, STR turns score a tool hit of 0.564 against 0.859 for the other fourth turns. On the corrected records, the prompt-length-adjusted gap is 29.4 points over 1,379 records. This locates difficulty in choosing the reporting action at the first opportunity, even with reference evidence in the history.

Reviewer_5gMLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STAR-Bench evaluates AML agents using 23 executable tools, 1,258 single-turn cases, and 50 multi-turn STR workflows across 28 open-weight configurations. It finds that single-turn tool accuracy poorly predicts workflow completion and identifies distinct reporting-related failures. The released code and saved evaluation records largely reproduce the headline findings, although independent inspection reveals limitations in the completion metric and an error affecting secondary STR-quality analysis.

Critical strengths
  1. Executable, domain-grounded benchmark with expert-verified cases: The benchmark provides 23 executable AML tools operating against a synthetic transaction database, with cases independently reviewed by domain experts for tool-role uniqueness, parameter observability, and difficulty consistency. Its reproducible environment and deliberately overlapping tool functions provide a meaningful evaluation resource for a domain where real regulatory data is difficult to release.
  2. Informative decomposition of multi-turn workflow failures: The executed-oracle and end-to-end evaluations distinguish errors in tool selection from failures caused by accumulated execution context. The finding that report-generation failures persist under oracle conditions, supported by paired statistical comparisons, provides more diagnostic insight than conventional aggregate tool-calling metrics.
Critical weaknesses
  1. The headline workflow-completion metric substantially overstates successful execution, while autonomous reporting decisions remain untested: Workflow completion requires the inclusion of the Gold tool at every turn but does not require correct arguments, absence of unnecessary calls, or a valid STR. An independent stricter calculation from the released scores reduces Mistral-Small-24B from 72% to 10% completion and Kanana-2-Think from 68% to 2%, demonstrating that the headline rankings are highly sensitive to the success definition. Furthermore, every STR-writing turn explicitly requests a report, so the benchmark does not evaluate whether agents independently recognize when regulatory reporting is required.
  2. Confirmed analysis errors and limited STR-grounding validation weaken confidence in some diagnostic claims: The raw-log loader in _experiments/scripts/analysis/load.py reads only the final segment of resumed runs, causing the STR-quality analysis to report zero generated reports for Qwen3.5-4B (non-thinking), despite 29 valid generate_str calls in the complete archived logs. Additionally, the STR checker relies on numerical overlap with earlier tool outputs without verifying correct transaction-level attribution, and its entity extraction does not recognize full 16-digit account identifiers. These limitations affect the reliability of secondary report-quality conclusions, even though the main tool-selection results reproduce correctly.
Questions for authors
  1. Can the authors report a strict end-to-end success metric requiring correct tool selection, valid arguments, no unnecessary calls, and a sufficiently complete and grounded STR? How would this affect the model rankings and the conclusion that Mistral-Small-24B achieves the highest workflow completion?
  2. The released raw-log loader appears to omit earlier segments of resumed evaluations. How do the authors reconcile the reported 0/45 generated STRs for Qwen3.5-4B (non-thinking) with the 29/45 generate_str calls in the archived logs, and which reported analyses change after correcting the loader?
  3. Since STR generation is explicitly requested, how does the benchmark distinguish regulatory-obligation recognition from ordinary instruction following? Are there experiments in which agents must independently determine whether reporting is necessary, including cases where no report should be generated?
  4. How reliably does the STR checker establish that reported figures belong to the correct accounts and transactions, rather than merely appearing somewhere in earlier tool outputs? Can the authors report transaction-level grounding accuracy or independently validated report correctness?
6 comments

Response to Reviewer 5gML (1/3)

Thank you for examining the released records. We corrected the loader and argument-binding errors. The revised PDF marks all revisions in blue. This comment answers W1 with Q1, the second answers W1 with Q3 and W2 with Q2, and the third answers W2 with Q4.

(W1 & Q1) A strict end-to-end success metric

We applied stricter reference-based checks, and the leading models changed. In Oracle, under a strict criterion that also requires the applicable arguments and no tool outside the reference at every turn, Gemma-4-31B and Qwen3.5-27B (NT) lead at 15 of 50 each, while Mistral-Small-24B completes 5.

However, Mistral-Small-24B has the highest c, at 36 of 50 in Oracle and 32 of 50 in end-to-end (E2E) runs. As you note, c checks reference tool selection or the specified no-call behavior at every turn, rather than a successful investigation. The table below compares Oracle results only.

Oracle scenarios completed (of 50)cStrict
Mistral-Small-24B365
Kanana-2-Think341
Gemma-4-31B3315
Qwen3.5-27B (NT)2815

Requiring correct arguments causes the largest drop. Each row below adds requirements, counted over configuration-scenario pairs. Oracle has 28 configurations and E2E has 25, each with 50 scenarios. Oracle supplies reference calls and outputs as history without executing model proposals, so the final execution check applies only to E2E.

Completion stageOracle (of 1,400)E2E (of 1,250)
Reference tool or specified no-call at every turn (c)538424
and applicable arguments, no tool outside the reference13389
and every annotated reference assessed, with error-free executionn/a88

E2E arguments without a retrieved source remain unassessed, so passing the argument checks differs from having every annotated reference assessed. Oracle and E2E also differ in history and argument checks, so their strict counts are not directly comparable.

Limiting repeated calls leaves the leaders at the final E2E stage unchanged. With or without this filter, Qwen3.5-27B (NT) and Qwen3.6-27B lead at 11 of 50 each, and Mistral-Small-24B has 8. Limiting reference-tool call counts lowers c passes from 424 to 415 and final-stage passes from 88 to 87. We did not judge whether the excluded repetitions were necessary. The report checker does not validate transaction attribution, so we did not use its score as a completion threshold (see Q4).

In 24 original E2E scenarios with fixed query constraints, across 25 configurations, we checked the facts in the last report at the requested turn. It is verified when it has a summary and every listed transaction gives date, sender, receiver and amount. Each such transaction must also match the database and earlier tool output and agree with the report's stored fields.

Original cVerified factual submissionNot verified
Pass15858
Miss82302

Of the 82 verified submissions with c misses, 62 miss only clarification turns. A configuration can therefore miss c and still submit correct facts, or meet c without a verified submission. Across these configurations, single-turn hit correlates at 0.81 with verified submission and at 0.67 with c. The check covers submitted facts, not the narrative or whether every required transaction is reported.

For each of 20 new factual goals, we built four tool routes. All 80 submit the required facts, but the 20 that use SQL aggregation instead of the profile tool fail a reference-tool check like c. These are routes we constructed, not alternatives the models found. An outcome check can thus credit equivalent routes while per-turn scoring keeps the steps the analyst requested, as the follow-up appendix describes.

In the Oracle setting, the strict criterion also brings completion closer to single-turn hit over the 28 configurations (Spearman's rho 0.85 compared with 0.55 under c). Within the band of ten configurations with similar single-turn hit, Spearman's rho between single-turn hit and completion is 0.79 under the strict criterion and −0.23 under c. Among these configurations, single-turn hit tends to predict completion with correct arguments but not tool-selection completion, and Section 4.4 now makes only this narrower claim. Tool-selection completion asks whether the agent takes every step the workflow requires, including the report. Correct arguments at other turns do not make up for a skipped step.

In the Abstract, the Figure 1 caption, Section 4.4 and the Conclusion we present c as tool-selection completion and no longer claim an overall best model. Section 4.4 reports the strict criterion next to c. Within that band, c spans 36% to 68%. The 32-point spread in the band is larger than scenario sampling alone produces (within-scenario permutation test, p = 0.006).

Response to Reviewer 5gML (2/3)

(W1 & Q3) Recognizing when a report is required

The main runs test whether an agent carries out a requested reporting step using earlier evidence, not whether it decides on its own that a report is due. Even when asked, models often fail to initiate that step at the first opportunity (Section 4.4). At the fourth turn, STR turns score a tool hit of 0.564 against 0.859 for the other fourth turns. On the corrected records, the gap adjusted for prompt length is 29.4 points over 1,379 records.

This design follows the agent's role in the source platform, where an analyst judges a case reportable and asks the agent to draft the report. FATF places the reporting obligation on the institution and calls for a compliance officer at management level (FATF 2012, Recommendation 20 and Interpretive Note to Recommendation 18). FATF also emphasizes manual review and human input when new technologies support AML/CFT work (FATF 2021, paragraph 7). The 45 report scenarios therefore test the requested step. Five scenarios ask whether an STR is needed and expect no report, but credit any nonempty reply without a tool call, so they check only that the model does not file one.

The additional test shows that report-tool initiation depends on the prompt for the two models tested. Mistral-Small-24B and Gemma-4-31B proposed generate_str in 39 and 33 of 45 cases when asked for a report, but in none of the 50 neutral questions over the same evidence, which include those 45 tasks. With an 8,192-token allowance and one round, we collected 110 new records and reused 90 requested-report controls. Those controls came from an earlier run of the appendix's evidence-presentation comparison, so the counts differ slightly from its table.

This tests prompt sensitivity of report initiation, not the accuracy of autonomous reporting decisions. Answer text may still contain a reporting judgment when no tool is called. We do not score that text because no expert has labeled whether a report is due under the neutral question. Section 4.3 separates requested reporting from autonomous initiation, and the appendix reports this test without claiming legal accuracy.

(W2 & Q2) The loader error and the analyses it changes

Your finding is correct. The 0 of 45 came from a loader that read only the last segment of resumed runs, and the corrected count is 29 of 45. A separate row-reference error overstated the argument gap. Combining compatible segments and rejecting duplicate turns recovers 219 turns for Qwen3.5-4B (NT). With the original extractor, its Grounding is 0.580 and its Overall 0.422. The corrections change the following results.

  • Oracle report quality: one of 28 rows changes and none of the six Table 3 rows does. Report-producing configurations increase from 25 to 26.
  • E2E execution: calls change from 4,434 to 4,654 and the error rate from 2.3% to 2.2%.
  • Fourth-turn reporting gap: usage records rise from 1,330 to 1,379, and the prompt-adjusted gap changes from 28.9 to 29.4 points.
  • Argument binding: we also fixed predict_fraud so that all six arguments bind to the row the model retrieved, leaving missing fields unassessable. The Oracle-to-E2E argument gap changes from 8.9 to 7.9 points, or from 6.4 to 5.9 points when only checks shared by both settings and both scorings are compared.

We corrected these figures and references in Sections 3.5 and 4.4 and in the appendices, and keep the effects of the two fixes apart. The corrections change no conclusion of the paper. Neither changes tool hit or tool-selection completion, as your review also found. End-to-end execution still lowers argument accuracy by 7.9 points against 2.2 for completion, so the Section 4.4 finding that it costs argument grounding more than tool selection stands.

References (both are cited in the paper)

  • FATF (2012). International Standards on Combating Money Laundering and the Financing of Terrorism & Proliferation. The FATF Recommendations, updated October 2025.
  • FATF (2021). Opportunities and Challenges of New Technologies for AML/CFT.

Response to Reviewer 5gML (3/3)

(W2 & Q4) Transaction-level grounding

You are right that numerical overlap cannot show that a figure belongs to the correct account or transaction. The submitted checker does not test attribution, so we added the checks below.

Three AML practitioners outside the author team, with 4, 5 and 3 years at financial institutions, rated the same 45 reports independently, without model names. They read the whole draft, its transaction list included, while the checker reads only the summary. Against the mean of their ratings, Spearman's rho is 0.66 for required fields, 0.58 overall, 0.32 for terminology and 0.24 for grounding, the last not significant. Each practitioner, set against the mean of the other two, exceeds the checker by 0.03 on average for required fields and 0.09 overall, but by 0.24 for grounding and 0.27 for terminology. All five raters order the three configurations on the overall score as the checker does. A practitioner flags misattribution in 5 of the 45 reports, and two agree on only one. At least two flag an important omission in 20, including 7 of the 18 with checker grounding 1. Appendix F now reports these ratings beside the authors'.

In the row-level check, a figure written in the same clause as an account or transaction identifier is correct when one preceding tool-output row holds both. Only 241 of 758 Oracle reports contain such a pair. Of their 595 checkable figures, 3.9% are paired with a different row, and a hand check confirms five of ten such flags. Another 12.6% are absent from the tool outputs, a distinct error category. Only 9 of the 45 rated reports are checkable. The check finds no wrong pairing in them while the practitioners flag three, so we report it as a sensitivity rather than a validated score.

To test the checks themselves, we built 632 controlled reports, each paired with its evidence and with the required transactions labeled by SQL. The reports are generated from fixed English and Korean grammars. Altered reports change the account, amount, direction or date of a transaction, omit a required transaction or add one. The table counts the reports each check would accept among the 504 held out, which use 16 factual goals kept out of development.

Required factsCasesFull numerical overlapRow check, no flagsTransaction and coverage pass
Supported and complete66666666
Misbound, missing or extra rows438438960

Every altered report keeps full numerical overlap, and 246 contain exactly the same numbers as the correct report. The row check still accepts 96, and only checking each full transaction and the coverage of required rows rejects all 438. Independent Python and SQL labels agree on all 632 reports. The controls show that these constructed defects can be detected on new transactions in two grammars, and the follow-up appendix now includes them.

Two LLM judges assessed 156 proposed summaries, including the original 45, and each flag cites the IDs of the report and evidence spans it rests on. The protocol checks the format and consistency of a judgment but not whether it is correct. Both judges passed 64 synthetic controls and eight ID-sensitivity pairs.

Of 156 summariesGemini 2.5 ProClaude Haiku 5.5
Protocol-valid judgments145137
Flagged a potential problem90129
Flagged, checker grounding 141 of 63 (65%)55 of 59 (93%)
Flagged, checker grounding below 149 of 82 (60%)74 of 78 (95%)

The judges flag summaries that the checker scores 1 about as often as the others, and they agree on 84 of the 127 summaries that both judged validly. The cross-check therefore does not show that the grounding score measures attribution.

You also note that the extractor misses full 16-digit account identifiers. It does, but counting them would not make the score measure attribution. On the corrected Oracle logs this changes 25 of 28 grounding rows, including all six Table 3 rows. It also lowers evidence-match agreement with the two author raters, from 0.728 to 0.625 and from 0.704 to 0.590, and leaves the practitioners' at 0.24. Two reports whose only figure is an account number move from 0 to full grounding, although both author raters scored them lowest.

In Section 4.3, Table 3 and the follow-up appendix we keep the original-extractor scores, recomputed after the loader correction. Section 3.5 now states that the checker reads only the summary. We report grounding with 16-digit account numbers counted, the row check and the LLM flags as separate sensitivities, and state that transaction-linkage accuracy is not validated. The reporting-turn drop in Section 4.4 and the no-call, fallback and keyword failures in Section 4.3 come from tool calls and do not depend on the checker.

We hope these analyses address your concerns, and we would be glad to clarify anything further.

Reviewer_P2XeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

STAR-Bench tests whether language models can use anti-money-laundering tools and prepare suspicious transaction reports (STRs). The benchmark consists of 23 tools across four subdomains, 1,258 single-turn cases, and 50 multi-turn scenarios, written and checked by AML practitioners. In the main evaluation, each new turn receives the correct previous calls and results, regardless of the model’s earlier actions. A separate experiment carries forward the model’s own results. The main finding is that models with similar accuracy on individual tool calls can perform quite differently on the reporting scenarios, with failures often occurring at report generation. This is supported by comparisons across 28 model configurations.

Critical strengths

The benchmark makes a gap in model evaluation concrete. Strong performance on individual tool calls does not reliably identify the models that perform best on the reporting scenarios: ten configurations with similar single-turn accuracy range from 36% to 68% in completing the required tool selections. The executable, expert-reviewed AML benchmark provides a concrete way to measure these differences.

Critical weaknesses
  1. The benchmark measures agreement with one prescribed workflow, rather than successful AML investigation

Cases specify expected tools, arguments, and their order, but expert verification does not establish that other investigative approaches are invalid. Following the reference calls also does not guarantee a correct result: completion ignores argument correctness, and the report checker can miss figures attributed to the wrong transaction. This limits what the rankings can claim about practical AML capability. The authors should demonstrate that valid alternative approaches receive credit and that successful outcomes require substantively correct evidence and reports. I believe rubric-based evaluation could be a promising path towards this.

  1. The main comparison mixes workflow difficulty with differences in tasks and evaluation rules

The main evaluation gives the model the correct previous calls and results before each turn, so it tests next-action prediction from corrected histories rather than a continuous investigation. It also introduces report generation only in the multi-turn setting and permits fewer tool-calling rounds. The ranking differences therefore do not establish whether conversation history, the additional task, or the execution limits explain the result. A matched comparison using the same tasks, evidence, and call budgets, alongside end-to-end outcome evaluation, would be informative.

Questions for authors
  1. How are valid investigations using different tools or a different order than the golden path scored? Have experts checked whether reasonable alternatives are rejected by the reference-based evaluation?
  2. Does the ranking disagreement persist when isolated report-generation requests receive the same evidence and call budget as the multi-turn tasks?
  3. In end-to-end evaluation, are later arguments checked against the records the model actually retrieved or against the fixed reference trajectory?
  4. How often do reports receiving high checker scores contain incorrect factual associations or important omissions when assessed by experts?
5 comments

Response to Reviewer P2Xe (1/4)

Thank you for asking what the scores say about valid investigations and correct reports. Your questions led us to correct how arguments are checked and to separate agreement with the reference from investigation success. The revised PDF marks all revisions in blue. This comment answers W1 and Q1, the second answers W2 and Q2, the third answers Q3, and the fourth answers Q4.

(W1 & Q1) Valid alternative investigations and completion

A replacement tool is credited only if annotated as valid. Call order within a turn does not affect tool hit or c, but moving a required action to another turn can lose credit. Experts reviewed the benchmark cases, but have not assessed whether the models' deviating investigations were valid alternatives.

A second AML expert checked tool roles, argument observability and difficulty in every case, and cases with several plausible tools were revised or removed (Section 3.3, Appendix D). Where a second tool remained valid, it is annotated and credited, which applies to 15 single-turn cases and no workflow turn.

This scoring evaluates execution of analyst requests. In the source AML platform, an analyst directs the investigation one request per turn, and the agent carries it out. The platform was designed with AML practitioners, and each benchmark tool runs its own platform code (Section 3.1). The first scenario requests transaction retrieval, risk prediction for the first transaction, one-hop network analysis and STR writing in sequence. Tools reading the same data can serve different regulatory roles, so a transaction query cannot replace a requested CTR-candidate detector (C1 in Section 1). The rankings therefore describe execution of analyst requests rather than independent investigation.

Requiring correct arguments causes the largest drop. Each row below adds requirements, counted over configuration-scenario pairs. Oracle has 28 configurations and E2E has 25, each with 50 scenarios. Oracle supplies reference calls and outputs as history without executing model proposals, so the final execution check applies only to E2E.

Completion stageOracle (of 1,400)E2E (of 1,250)
Reference tool or specified no-call at every turn (c)538424
and applicable arguments, no tool outside the reference13389
and every annotated reference assessed, with error-free executionn/a88

The appendix states that these stages do not measure semantic STR success.

In 24 original E2E scenarios with fixed query constraints, across 25 configurations, we checked the facts in the last report at the requested turn. It is verified when it has a summary and every listed transaction gives date, sender, receiver and amount. Each such transaction must also match the database and earlier tool output and agree with the report's stored fields.

Original cVerified factual submissionNot verified
Pass15858
Miss82302

Of the 82 verified submissions with c misses, 62 miss only clarification turns. A configuration can therefore miss c and still submit correct facts, or meet c without a verified submission. Across these configurations, single-turn hit correlates at 0.81 with verified submission and at 0.67 with c. The check covers submitted facts, not the narrative or whether every required transaction is reported.

For each of 20 new factual goals, we built four tool routes. All 80 submit the required facts, but the 20 that use SQL aggregation instead of the profile tool fail a reference-tool check like c. These are routes we constructed, not alternatives the models found. An outcome check can thus credit equivalent routes while per-turn scoring keeps the steps the analyst requested, as the follow-up appendix describes.

Rubric-based evaluation, as you suggest, would extend these checks to the narrative.

In Oracle tool-selection completion, the ten configurations with similar single-turn hit still differ by 32 points. The strict criterion requires the applicable arguments and no tool outside the reference at every turn. Within the band of ten configurations with similar single-turn hit, Spearman's rho between single-turn hit and completion is 0.79 under the strict criterion and −0.23 under c. Among these configurations, single-turn hit tends to predict completion with correct arguments but not tool-selection completion, and Section 4.4 now makes only this narrower claim. Tool-selection completion asks whether the agent takes every step the workflow requires, including the report. Correct arguments at other turns do not make up for a skipped step.

Response to Reviewer P2Xe (2/4)

(W2 & Q2) The reporting difference under matched tasks, evidence and budgets

Yes for one-round report proposals, but not for verified factual submissions. With tasks, evidence, tools, output allowance and input length matched, Mistral-Small-24B leads Gemma-4-31B on dialogue proposals but trails it on isolated proposals. Gemma-4-31B leads Mistral-Small-24B on verified facts in both presentations. The comparison covers the four configurations in the table below, not a reranking of all 28.

The original-log controls show that neither the round limit nor the report task alone explains the completion differences. Limiting single-turn cases to one round, as workflow turns are limited, does not remove them. Scoring single-turn cases on their first round only leaves the correlation with c at 0.57 compared with 0.55, and within the band first-round hit still does not order c (−0.18). Its correlation with the strict criterion in the band falls from 0.79 to 0.44. The report task accounts for part of the gap but not all of it. Removing the report turn raises the correlation with single-turn hit from 0.55 to 0.73, but the ten configurations with similar single-turn hit still span 26 points in completion and single-turn hit does not order them (0.22). Conversation history does change reporting for some configurations, as the matched comparison shows. This is the workflow behavior the benchmark is built to measure, and an isolated request cannot show it.

You note that the Oracle setting tests next-action prediction on corrected histories rather than a continuous investigation. We reset the history to the reference on purpose, so that each failure is attributed to the turn where it occurs, and the end-to-end runs test the continuous investigation separately (Section 4.4). In those runs, the nine configurations of the band with an end-to-end run span 34 points in completion, from 28% to 62%, and single-turn hit does not order them either (0.21), as the follow-up appendix reports.

You are right that the submitted comparison did not hold task and budget fixed, so we ran the matched comparison you suggested. Mistral-Small-24B, Gemma-4-31B and both settings of Qwen3.5-27B, plus Qwen3.5-4B (NT) and a repeat of Gemma-4-31B, received the same 45 report requests and evidence, identical tools and a shared 8,192-token output allowance per task, with one or five rounds. The allowance is below the 16,384 tokens of the main runs. We matched the initial input length by padding the dialogue presentation, which is shorter, to the provider-reported token count of the isolated one. Including the unpadded runs, this gave 1,620 traces over all 45 tasks, and traces that share a long input were treated as dependent.

For the first four configurations, the table counts nonempty generate_str proposals and verified factual submissions out of 45 at matched length, dialogue first. Verification uses the checks of our answer to W1, without the query constraints that exist for only 24 tasks. Neither count assesses the narrative.

ConfigurationProposals, 1 roundProposals, 5 roundsVerified, 1 roundVerified, 5 rounds
Mistral-Small-24B40 / 2545 / 2426 / 2133 / 19
Gemma-4-31B31 / 3145 / 4528 / 2540 / 39
Qwen3.5-27B (NT)29 / 1645 / 4127 / 1542 / 40
Qwen3.5-27B (T)29 / 2245 / 4428 / 2143 / 42

Input length alone does not explain the proposal differences. At one round, the dialogue presentation adds 33.3 points of proposals for Mistral-Small-24B and 28.9 for Qwen3.5-27B (NT), but none for Gemma-4-31B. Over 10,000 paired task resamples, their 95% intervals are 20.0 to 48.9 and 13.3 to 44.4.

In verified submissions, the one-round dialogue advantage is 11.1 points for Mistral-Small-24B and 26.7 for Qwen3.5-27B (NT), with paired 95% intervals of −4.4 to 26.7 and 13.3 to 40.0. Gemma-4-31B then leads Mistral-Small-24B in both presentations, by two and four of the 45 tasks. Of the 40 dialogue reports of Mistral-Small-24B, 13 have a missing or incomplete transaction list, so their facts cannot be checked.

With five rounds, the isolated counts of Qwen3.5-27B come close to its dialogue counts, while Mistral-Small-24B keeps a large gap. All four configurations then propose the report in every dialogue task. Workflow turns allow one round because the Oracle setting scores a proposal without executing it, so a second round would have no tool output to act on. The one-round score measures whether a model reports at its first opportunity, and Section 4.4 says so.

We added this comparison and its interpretation to Section 4.4 and the appendix, and report initiation and verified submission separately there, because the dialogue advantage of Mistral-Small-24B is smaller in verified submissions. Padding equalizes token count but not how the evidence is organized, and we did not rerun the full 28-configuration ranking.

Response to Reviewer P2Xe (3/4)

(Q3) Arguments checked against the retrieved row

We now check arguments against the model's retrieved rows, not a fixed reference row. Correcting one binding error in fraud prediction restores full credit to 89 turns. Thank you for exposing it. All 46 annotated context references already followed the model's own output, but fraud prediction bound only its amount to the retrieved row and five unannotated arguments to the reference row. This penalized calls whose arguments all matched the retrieved row. All six arguments now follow the retrieved row in all 15 references of this tool, and missing fields remain unassessable.

Of the 205 affected turns with complete rows, 198 reproduce the full argument tuple, six omit the call and one has incorrect arguments. Across 25 E2E configurations, the 375 affected turns comprise these 205 complete rows, 17 partial rows and 153 unavailable rows.

The Oracle-to-E2E argument gap changes from 8.9 to 7.9 points, or from 6.4 to 5.9 points when only checks shared by both settings and both scorings are compared. We corrected the rule in Section 3.5 and the figures in Section 4.4 and the appendix. The correction makes argument scoring consistent with the retrieved evidence and leaves tool hit and tool-selection completion unchanged. End-to-end execution still lowers argument accuracy by 7.9 points against 2.2 for completion, so the Section 4.4 finding that it costs argument grounding more than tool selection stands.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.