- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces STAR-Bench, a compliance-oriented benchmark for evaluating LLM agents on anti-money laundering (AML) regulatory reporting workflows. The authors argue that AML investigations differ from general tool use along four axes: overlapping tool roles (C1), a mandated reporting step (the suspicious transaction report, STR) that must be scored as an action rather than prose (C2), evidence that must carry across turns (C3), and model selection driven by proxies (function-calling leaderboards, specialization labels, schema language, reasoning mode) whose bearing on this workflow is unknown (C4).
Critical strengths
-
A benchmark design that measures a mandated reporting step as an action, with execution-grounded evaluation throughout. Scoring the STR step as a tool call (C2), grading the produced report against the outputs of the turns before it, executing every tool against a real database, and injecting executed gold outputs in the oracle setting together produce an evaluation in which failure attribution is clean: the paper can show the workflow breaks specifically at the report-writing turn (0.564 vs 0.859 hit at turn 4), surviving an oracle that erases all earlier mistakes. This is consequential because it converts a vague practical worry ("agents can't do compliance") into a located, reproducible measurement — and none of BFCL, ToolSandbox, τ-bench, or the financial benchmarks score anything like it (Table 1).
-
A rigorously supported negative result with direct deployment relevance: proxies do not order agents on this workflow. The claim that single-turn hit does not resolve workflow readiness is backed by a concrete, statistically tested spread (Δc = 0.32 within a 7-point single-turn band; paired bootstrap [16,44] and [20,52] with McNemar p ≤ 0.0007), and the same discipline is applied to BFCL rank correlation, specialization pairs, the 2×2 language ablation, and thinking/non-thinking pairs. This matters because model selection for regulated on-premises deployment is done exactly by these proxies; showing their unreliability on a benchmark built from the actual regulated workflow is the kind of evidence that changes procurement and evaluation practice, and it makes the benchmark useful even to groups that never touch AML.
Critical weaknesses
- All workflow-level conclusions rest on 50 scenarios (only 45 ending in an STR turn; 22 long-context, 14 base, 14 missing-parameter). Completion is a binary per scenario, so per-configuration c estimates have ±10+ point uncertainty, and the finding that "the best completer ranks 17th on single-turn hit" — a centerpiece of Figure 1 and the abstract — depends on the particular scenario mix. The authors test differences between selected configuration pairs, but no scenario-count or scenario-composition sensitivity analysis is reported (e.g., does the Δc = 0.32 spread survive leave-one-fraud-type-out?). This is consequential because the paper's main claim is precisely that workflow-level evaluation reveals what single-turn evaluation misses; the claim is only as sturdy as the 50 scenarios.
- The STR checker is keyword/regex-based; its "grounding" dimension checks that a figure appeared in some earlier tool output without verifying the figure is bound to the correct account/transaction, and its validation (Appendix F) uses two raters who are authors, on 45 reports, with a floor effect on the terminology dimension (much of the evidence-match agreement comes from the shared zero convention — dropping no-figure reports leaves ρ ≈ 0.33–0.42). Since "writing an STR grounded in the evidence" is one of the benchmark's headline capabilities (C3/C2), the community will cite these checker scores; the current validation is not strong enough to support that weight, and an independent (non-author) rater study or an LLM-as-judge cross-check would materially change confidence in Table 3.
Questions for authors
-
In the executed-oracle setting, after a turn where the model deviates from the gold call, the next turn receives the gold call's executed output. How do you handle cases where the model's earlier deviation makes the gold next call semantically incoherent for the model (e.g., it never retrieved the account the gold Turn-3 call references)? Are such scenarios excluded, scored as failures, or does the injected context simply "reset" the investigation? This bears directly on how the turn-4 drop should be interpreted.
-
Can you report a sensitivity analysis of the headline workflow claims to scenario composition — e.g., the Δc spread and the best-completer ranking under leave-one-scenario-type-out or bootstrap over the 50 scenarios? With c binary over n = 50, what is the confidence interval on Mistral-Small-24B's c = 0.72 itself?
-
The reasoning-mode conclusion (Section 5.3) rests on four thinking/non-thinking pairs and an explicit output budget, with budget exhaustion as the stated mechanism. What was the budget, and does the −2.8 / −7.5 effect persist at a larger budget? If the effect is budget-bound, the claim should be scoped accordingly in the abstract, where "reasoning mode lowers them on average" currently reads as budget-free.
-
For missing-parameter multi-turn scenarios, how exactly is a "correct clarification" scored — is any no-call response credited, or must the model ask for the specific missing arguments? A generous rule here would inflate turn-level hit.
7 comments
Response to Reviewer B6bj (2/4)
(W2) Evidence attribution and checker validation
You are right that numerical overlap cannot show that a figure belongs to the correct account or transaction. We ran both checks you suggest, ratings by practitioners outside the author team and an LLM cross-check, and added a row-level check and the controlled tests below. The two author raters give overall correlations of 0.865 and 0.827, but as you noted, their evidence-match correlations fall to 0.417 and 0.326 on the 32 summaries with extracted figures, and their terminology scores sit near the floor.
Three AML practitioners outside the author team, with 4, 5 and 3 years at financial institutions, rated the same 45 reports independently, without model names. They read the whole draft, its transaction list included, while the checker reads only the summary. Against the mean of their ratings, Spearman's rho is 0.66 for required fields, 0.58 overall, 0.32 for terminology and 0.24 for grounding, the last not significant. Each practitioner, set against the mean of the other two, exceeds the checker by 0.03 on average for required fields and 0.09 overall, but by 0.24 for grounding and 0.27 for terminology. All five raters order the three configurations on the overall score as the checker does. A practitioner flags misattribution in 5 of the 45 reports, and two agree on only one. At least two flag an important omission in 20, including 7 of the 18 with checker grounding 1. Appendix F now reports these ratings beside the authors'.
In the row-level check, a figure written in the same clause as an account or transaction identifier is correct when one preceding tool-output row holds both. Only 241 of 758 Oracle reports contain such a pair. Of their 595 checkable figures, 3.9% are paired with a different row, and a hand check confirms five of ten such flags. Another 12.6% are absent from the tool outputs, a distinct error category. Only 9 of the 45 rated reports are checkable. The check finds no wrong pairing in them while the practitioners flag three, so we report it as a sensitivity rather than a validated score.
To test the checks themselves, we built 632 controlled reports, each paired with its evidence and with the required transactions labeled by SQL. The reports are generated from fixed English and Korean grammars. Altered reports change the account, amount, direction or date of a transaction, omit a required transaction or add one. The table counts the reports each check would accept among the 504 held out, which use 16 factual goals kept out of development.
| Required facts | Cases | Full numerical overlap | Row check, no flags | Transaction and coverage pass |
|---|---|---|---|---|
| Supported and complete | 66 | 66 | 66 | 66 |
| Misbound, missing or extra rows | 438 | 438 | 96 | 0 |
Every altered report keeps full numerical overlap, and 246 contain exactly the same numbers as the correct report. The row check still accepts 96, and only checking each full transaction and the coverage of required rows rejects all 438. Independent Python and SQL labels agree on all 632 reports. The controls show that these constructed defects can be detected on new transactions in two grammars, and the follow-up appendix now includes them.
Two LLM judges assessed 156 proposed summaries, including the original 45, and each flag cites the IDs of the report and evidence spans it rests on. The protocol checks the format and consistency of a judgment but not whether it is correct. Both judges passed 64 synthetic controls and eight ID-sensitivity pairs.
| Of 156 summaries | Gemini 2.5 Pro | Claude Haiku 5.5 |
|---|---|---|
| Protocol-valid judgments | 145 | 137 |
| Flagged a potential problem | 90 | 129 |
| Flagged, checker grounding 1 | 41 of 63 (65%) | 55 of 59 (93%) |
| Flagged, checker grounding below 1 | 49 of 82 (60%) | 74 of 78 (95%) |
The judges flag summaries that the checker scores 1 about as often as the others, and they agree on 84 of the 127 summaries that both judged validly. The cross-check therefore does not show that the grounding score measures attribution.
In the main text the checker scores appear only in Table 3 and its paragraph in Section 4.3, and neither the Abstract nor the Conclusion uses them. C3 is tested through the arguments of the multi-turn scenarios, and the reporting step of C2 is scored as a tool call, as your first strength notes.
In Section 4.3, Table 3 and the follow-up appendix we keep the original-extractor scores, recomputed after the loader correction. Section 3.5 now states that the checker reads only the summary. We report grounding with 16-digit account numbers counted, the row check and the LLM flags as separate sensitivities, and state that transaction-linkage accuracy is not validated. The reporting-turn drop in Section 4.4 and the no-call, fallback and keyword failures in Section 4.3 come from tool calls and do not depend on the checker.
Response to Reviewer B6bj (3/4)
(Q1) What the Oracle setting does after a deviation
Oracle resets the history to the reference calls and outputs after every turn, including after a deviation. No scenario is excluded. The model's deviating call is not executed or carried into the next turn, and it is scored as a miss at that turn. Thus the account referenced by a gold Turn-3 call appears in the history through executed reference outputs, even if the model never retrieved it. Oracle measures next-action selection on the reference path, while end-to-end (E2E) runs test the consequences of the model's own decisions. Section 4.4 now states this handling.
The reporting-turn drop persists with reference history at every turn, so it is not accumulated from earlier deviations. At the fourth turn, STR turns score a tool hit of 0.564 against 0.859 for the other fourth turns. On the corrected records, the prompt-length-adjusted gap is 29.4 points over 1,379 records. This locates difficulty in choosing the reporting action at the first opportunity, even with reference evidence in the history.
Response to Reviewer B6bj (1/4)
Thank you for the detailed questions on scenario composition, evidence attribution and reasoning mode. They led us to add sensitivity analyses and to state the scope of several claims more precisely. The revised PDF marks all revisions in blue. This comment answers W1 and Q2, the second answers W2, the third answers Q1, and the fourth answers Q3 and Q4.
Yes. The completion spread survives every tested scenario-type and fraud-type removal, while the leading configuration changes with the scenario type. The data thus support a difference between configurations but not a single best one. Mistral-Small-24B's c is 36 of 50, with a nominal Wilson 95% interval of 58.3% to 82.5%.
Leaving out one fraud type at a time, as you suggest, keeps the spread between 30.0 and 41.7 points. Mistral-Small-24B stays first in all six cases, tied in two. The table removes one scenario type at a time. The spread is over the fixed band of ten configurations with similar single-turn hit, and the rank over all 28.
Paired comparisons put the spread beyond scenario sampling alone. The within-scenario permutation test gives p = 0.006 for the 32-point spread, with a null 95th percentile of 26 points. It keeps the ten configurations paired on the same scenarios, and each scenario-type removal gives p below 0.02. Your concern about uncertainty above ten points holds for one configuration's c, but the paired intervals you cite, [16, 44] and [20, 52] points, exclude zero.
Two band configurations share the lowest c. One of them, Qwen3.5-27B (T), completes 48% when rerun in a separate collection, so Section 4.4 now compares the leaders with the other, Qwen3.6-35B-A3B. The paired intervals become [18, 42] and [20, 52].
The identity of the winner also varies under resampling. The Wilson interval is not corrected for selecting the winner. In 10,000 paired resamples of the 50 scenarios over the 28 configurations, Mistral-Small-24B is uniquely first in 44.5% and first including ties in 56.2%. Both are resampling frequencies rather than probabilities that it is better in the population.
We removed the rank of the best completer from the Abstract, and Figure 1, Section 4.4 and the Conclusion now state that the leading configuration changes with the scenario mix. The strict criterion requires the applicable arguments and no tool outside the reference at every turn. Its spread in the band also exceeds scenario sampling (p = 0.001). Within the band of ten configurations with similar single-turn hit, Spearman's rho between single-turn hit and completion is 0.79 under the strict criterion and −0.23 under c. Among these configurations, single-turn hit tends to predict completion with correct arguments but not tool-selection completion, and Section 4.4 now makes only this narrower claim. Tool-selection completion asks whether the agent takes every step the workflow requires, including the report. Correct arguments at other turns do not make up for a skipped step.