- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds a safety benchmark for financial agents, with 877 executable cases covering customer service, internal assistance, and autonomous asset management. It looks at what the agent actually does with its tools, the current business state, and the relevant permission rules to check both whether it breaks a rule and whether it completes a legitimate task. Across different models, agent frameworks, and defenses, the results vary quite a bit by setting. One interesting finding is that an agent may refuse an unsafe action at first but go on to violate the rule later in the conversation.
Critical strengths
-
I appreciate the authors’ effort to study the safety of financial agents in practical business settings. I find the proposed benchmark a valuable contribution, particularly because it considers both safety and task utility. An agent that avoids violations by refusing every request would have limited practical value, and the distinction between violations, task completion, and safe completion captures this concern well.
-
Another strength is the use of actual execution traces and business states to evaluate safety, together with coverage of different deployment settings and financial scenarios. The experiments also offer useful insights into how agent frameworks and added defenses affect both safety and legitimate task completion, and how initially safe behavior can break down in later interactions.
Critical weaknesses
-
Although the paper evaluates nine models, they come from only three model families. Including other major families, such as Claude and Gemini, would help show whether the findings generalize beyond the tested models.
-
The threat model could be explained more clearly: who is the attacker, what can they control, and can they adapt based on the agent’s feedback? It would also help to distinguish direct requests for prohibited actions from indirect prompt injection, and provide a breakdown of attack strategies and input surfaces. Broad business coverage alone does not establish attack diversity.
-
Attacks and defenses are often a cat-and-mouse game: defenses are designed for particular threats and need the right information to detect them. Here, many violations depend on permissions or financial limits, which the tested defenses do not explicitly check. Their limited effectiveness may therefore reflect this mismatch. Could the authors include a defense that checks business rules and current state before execution?
-
In the record-update study, the underlying state changes without notifying the agent. It is unclear whether agents are told that records may change and should be checked again. Adding conditions with an explicit re-check instruction or a trusted update notification would help distinguish failures to seek fresh information from failures to use it correctly.
Questions for authors
-
I found the comparison with recent finance-specific agent benchmarks somewhat limited. In particular, ReguSim (https://arxiv.org/abs/2608.19974) also evaluates agent actions against executable financial rules and distinguishes attempted actions from enforced outcomes, while Fiducia-bench (https://arxiv.org/abs/2608.16055 ) uses machine-checkable policies and execution traces to study compliant behavior in financial workflows. Could the authors clarify more explicitly what evaluation capability this benchmark adds beyond these settings? I see the broader coverage of asset-management roles and business scenarios as a potential distinction, but would like to understand whether there is also a methodological difference.
-
The paper shows that the choice of agent framework can substantially affect safety. Why were Hermes and OpenClaw chosen, and how representative are they of financial agent deployments? It would help to clarify how much the findings depend on these particular frameworks.