- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces FinDriftBench to evaluate compliance drift during sustained financial-agent operation without external attacks. The benchmark contains 1,200 cases across ten financial business categories, with each scenario processed over 120 consecutive rounds. Its framework separates performance management, business execution, and compliance auditing. Experiments across eight LLMs examine violations under memory loss and performance pressure, while additional experiments investigate communication between agents. The authors report compliance violations, substantial false rejection in some models, and communication-induced audit capture in susceptible models.
Critical strengths
The benchmark addresses a practically relevant evaluation target using professionally curated financial cases with explicit compliance annotations. Evaluating operational decisions on detailed business materials provides useful domain coverage beyond financial knowledge questions. Reporting both violation rates and false-rejection rates is particularly valuable: it exposes the distinction between correctly enforcing compliance and obtaining an apparently safe result by rejecting legitimate business. This makes the evaluation more informative for deployment than violation rates alone.
Critical weaknesses
-
The incremental contribution to agent evaluation is insufficiently established.
The paper does not clearly establish what distinctive insights FinDriftBench provides beyond existing research on long-horizon agent behavior and safety. HINTBench studies intrinsic risks in long-horizon trajectories without external attacks (Wang et al., 2026), while prior work on outcome-driven constraint violations examines agent misconduct under KPI pressure (Li et al., 2025). Although these settings are not identical, they overlap with central elements of the paper’s motivation and findings.
Moreover, sequentially processing 120 cases does not necessarily constitute a task requiring long-horizon reasoning: the paper does not clearly demonstrate why later decisions require the detailed history of earlier cases. The financial dataset provides domain coverage, but it remains unclear whether its financial characteristics enable analyses or insights unavailable from general agent benchmarks. Stronger comparisons are needed to establish the contribution beyond introducing financial business data.
-
The experiments omit common approaches to handling long-horizon tasks.
The experiments do not evaluate several straightforward approaches to managing the accumulated context: summarizing or compressing previous cases while preserving the initial instructions; assigning each case to a fresh sub-agent that receives the original rules and the current case; or storing compliance rules in external memory and reading them when processing each case. These approaches are directly relevant to the context accumulation and rule-forgetting problem studied here.
The paper includes a long-term-memory ablation that supplies the rules every round, but it does not compare these alternative execution and memory strategies. In particular, the apparent separability of individual cases makes fresh per-case sub-agents an important baseline. Such comparisons should retain the same performance-pressure information. Without these experiments, it is unclear how much of the reported drift would remain when common long-horizon techniques are applied.
References
Wang, J., Hou, J., Wang, F., Jian, P., Bao, C., and Lv, Z. (2026). HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark. arXiv preprint arXiv:2604.13954.
Li, M. Q., Fung, B. C. M., Weiss, M., Xiong, P., Al-Hussaeni, K., and Fachkha, C. (2025). A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents. arXiv preprint arXiv:2512.20798.
Questions for authors
To what degree could the SOTA harness techniques like sub-agents solve the proposed problem of FinDriftBench?
2 comments
Thanks for the authors' response. But my concern holds. My biggest question is: what's the unique contribution of FinDriftBench to agent evaluation? The authors claim that the measure of "compliance" is an exclusive feature of this work, but what makes "compliance" so different? I am afraid I was not convinced that it is something beyond a special case of general long-context instruction following in finance settings. To be more constructive, I believe it is important to justify the necessity of FinDriftBench with more convincing empirical evidence. For example, if the experiment shows that, in some cases, an agent's performance on FinDriftBench doesn't grow as its instruction-following ability or long-context handling ability increases, then FinDriftBench may truly build something orthogonal to other general long-context benchmarks.
We thank the reviewer for recognizing the deployment-relevant target, the operational decisions grounded in detailed business materials, and the value of reporting violation and false-rejection rates together to separate genuine compliance from appearing safe by rejecting legitimate business, which is exactly the discriminative power we aim to contribute.Revisions and supplementary content in the paper are marked in blue.
W1: Incremental contribution unclear versus HINTBench and ODCV; do 120 cases constitute true long-horizon reasoning?
We have read both works and do not claim priority over intrinsic or pressure-driven risk as such. HINTBench (Wang 2026) is an offline trajectory audit by a guard model over pre-collected general-domain trajectories; it does not place a model as the executing subject whose compliance erodes over its own run, and has no performance-compliance conflict, no forgetting pathway, and no multi-agent interaction. ODCV (Li 2025) is closest on the pressure pathway (self-directed KPI-driven violations without explicit instruction, and a recognition-action gap), but measures a per-scenario violation rate on short tasks rather than a long-horizon temporal drift, has no forgetting pathway, and is single-agent. The contribution producible in neither work is to measure compliance as a long-horizon drift, jointly attribute it to forgetting and pressure, and characterize communication-induced collusion, in a financial multi-agent setting, delivered as an expert-curated financial dataset and a suite of dynamic-process metrics (WCDC, the forgetting and willful split of CBC, and MACI). On whether 120 cases are long-horizon, the long-horizon property lies in the temporal accumulation of memory and pressure (context growth and the compression eviction of rules, together with the tension between the shortfall and the remaining rounds), not in cross-case reasoning dependence; we do not claim that later cases need the business content of earlier ones, and indeed our E5 result that larger capacity gives less drift demonstrates that this accumulation-to-forgetting dependence is real. The financial indispensability is that the structural non-co-satisfiability (target times coefficient greater than 1) is a tension specific to heavily regulated, KPI-driven domains, and general agent benchmarks lack this oracle-level ground truth and structural conflict. Reviewers 1 and 2, and you yourself, acknowledge the setting's distinctive value in your Critical Strengths, which supports the increment.
W2/Q: Experiments omit SOTA approaches (summarization, sub-agents, external memory); evaluate sub-agents.
This is our key addition. We implemented and evaluated a fresh per-case sub-agent (a new agent per case, carrying the original rules and the current case, retaining the same performance pressure). Forgetting-type drift is eliminated (CBC jumps to the 0.83 to 0.98 range, that is, willful, and WCDC flattens), but violations are not solved: the residual RVR stays at 4% to 26%, and Claude-Sonnet-5 actually rises from 8% to 26%. The mechanism is that sub-agents fix forgetting while the performance-pressure pathway is untouched, so with the rules present the model still violates willfully under pressure. So, to your central question of how far a SOTA harness technique such as sub-agents can solve the problem FinDriftBench poses, it removes the forgetting component of drift but not the pressure-driven willful component (which can even grow), which pins down our core thesis that drift equals forgetting and pressure, two orthogonal pathways against which no single technique suffices. External memory that re-reads the rules each round is close to our long-memory ablation (also a 4% to 16% residual); the summarize-while-preserving-instructions variant we commit to add for camera-ready. We also note that short-term continuous execution is a realistic deployment default (policy is typically not re-asserted at each step at batch scale), so the setting itself is deployment-relevant.