acceptodds
Under review as a conference paper at ICLR 2027

LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Live Financial Markets

Abstract

Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Five frontier LLMs operate along continuous trajectories under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. We evaluate them through both realized outcomes and mechanism-specific diagnostics derived from complete decision traces. Across 30 days of live evaluation, we find a pronounced outcome–capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use. Trace-level diagnostics further expose distinct bottlenecks across capabilities, demonstrating that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability. LiveMACEBench makes this distinction measurable, turning live markets from a performance leaderboard into a diagnostic environment for agent capability.

Primary area:
datasets and benchmarks
Keywords:
Agent benchmark, long‑horizon evaluation, dynamic environment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_ZDQLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to evaluate LLM agents that run over long periods in changing real-world environments, using live financial markets as the test setting. The main contribution is LiveMACEBench. Five frontier LLMs paper-trade crypto and U.S. stocks for about 30 days under five matched setups: Base ReAct, Tool Use, Persistent Memory, Rule Following and Multi-Agent Collaboration. The authors score each setup with both returns and process metrics taken from full decision traces, using LLM judges and rule checkers. They report an "outcome-capability gap": return rankings change often over time, capability rankings stay stable, and similar returns can come from very different behavior.

Critical strengths

1- A clear and useful framing. The paper separates three things: having a mechanism, using it well, and benefiting from it (Sec. 1). The trace diagnostics support this. For example, Gemini stores 93 memories but has the lowest memory usage score (0.497), while GPT-5.4 retrieves memories 332 times but stores only one (Table 1).

2- Matched setups with full traces. All agents share one ReAct scaffold and the same live market, and every decision trace is recorded (Sec. 2). This lets readers locate specific failure modes, such as the concentration-limit errors in rule following (Sec. 3.2.3).

Critical weaknesses

1- The main "gap" result is an unfair comparison. Capability scores are running averages over many traces, which settle down by construction, while cumulative returns depend on the market path (Appx. C.7). So 0 vs 21 ranking reversals (Fig. 2a) is what you would expect statistically. It does not show that the capability metrics are more valid.

2- One run per setup, with no uncertainty. Each model-configuration pair runs once over one market window. Two runs failed and were dropped (Appx. C.3, C.5). Return and drawdown gaps of a few points are reported without error bars. For example, a +0.06 change in max drawdown is counted as a memory "improvement" (Table 1). The main tool-use judge is Qwen3.5, from the same family as one of the evaluated models (Fig. 3).

Questions for authors

1- (Clarification) Are capability rankings still more stable than return rankings if both use the same window? Examples would be daily capability scores against daily returns, or running-average returns.

2- (Clarification) Can you add repeated runs or a second market window for at least one capability, so that the setup effects have uncertainty estimates? 3- (Clarification) Did you validate the LLM judge scores against human ratings on a sample of traces? Does the Qwen judge give higher scores to Qwen3-Max?

Reviewer_2aJjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LiveMACEBench evaluates LLM agents through both trading outcomes and process-level measurements. Five model backbones run persistent paper-trading accounts using live cryptocurrency and U.S. stock data for about 30 days. A common ReAct setup is extended with tool routing, persistent memory, hierarchical rules, or multi-agent collaboration. The benchmark measures tool execution, memory construction and use, rule compliance, and observable collaboration. The main finding is that return rankings do not consistently agree with these process scores, and that adding a mechanism does not necessarily lead to effective use or better returns.

Critical strengths
  1. Looking beyond trading returns is useful. The separate measurements of rule execution and auditability, and of memory construction and use, help describe failures that a return leaderboard would hide.
  2. The paper provides detailed runtime descriptions, evaluation prompts, cross-judge checks, and concrete decision traces. Cases such as repeated reuse of one memory and incorrect post-trade exposure calculations give useful evidence about agent behavior.
Critical weaknesses
  1. The additional contribution over closely related benchmarks is not clear enough. Agent Trading Arena already studies live multi-market agents with different architectures, while CLQT explicitly studies diagnostic evaluation beyond returns. The paper cites these works, but does not sufficiently demonstrate what additional capability distinctions its evaluation can reliably establish.
  2. Some capability scores mainly measure observable activity. Role Diversity rewards calling more specialist roles, while Evidence Integration relies on source coverage and citation in the final decision. These measurements do not establish that the additional evidence is correct, necessary, or influential.
  3. The mechanism-effect claims rely on one roughly 30-day market path. No clearly budget-matched single-agent controls are reported. Memory, tool routing, and collaboration also change prompts, available information, and computation.
  4. Run exclusions need to be reflected in the reliability evaluation. Excluding the corrupted Gemini run caused by an environment defect is understandable. The Grok collaboration run, however, failed through repetitive tool calls and timeouts. This is relevant agent behavior and should appear as a completion or reliability result rather than only as a missing entry.
  5. The paper-trading execution model is not specified sufficiently. The rules mention approximate fee and slippage costs, but the manuscript does not clearly explain how these enter actual fills and account equity.

After rebuttal: The completion results and the clarification of trading costs address two of my concerns. The human checks are useful, but they do not cover the collaboration score. It is still unclear whether that score distinguishes useful contributions from simply involving more roles or citing more sources. The second market window only tests tool-use scores for two models. I keep my rating at 2.

Questions for authors
  1. What diagnostic conclusions does LiveMACE establish beyond exsiting methods? Can the same evaluation distinguish failure modes in existing complete financial-agent systems?
  2. How well do the process scores agree with independent human assessments? Do they respond correctly when evidence citations, memory content, or specialist contributions are changed while superficial trace structure is preserved?

Comments on the paper

1 comment
Area_Chair_LE5TOpenReview ↗

Review

Summary: LiveMACE evaluates five LLMs in a 30-day live paper-trading study, examining tool use, persistent memory, rule following, and multi-agent collaboration. It combines trading outcomes with process diagnostics to show that profitability and observable agent behavior can give different assessments of the same model.

Strengths: Distinguishing access to a mechanism, effective use, and eventual benefit is valuable. The paper clearly motivates why trading returns alone cannot explain an agent’s strengths and limitations.

The traces reveal meaningful differences between retrieving and using memories, following rules and explaining them correctly, and consulting specialists and acting on their advice.

Five model families and four capability-focused configurations operate over persistent accounts during the same market period. This provides a useful foundation for comparative analysis.

Weaknesses:

The broader findings still lack independent replication. The July replay supports the tool-use ordering of two models across a second window. However, it does not establish the reliability of the reported return and risk differences across configurations. The results should therefore remain framed as observations about these configurations and evaluation periods.

Human agreement provides useful support for tool use, memory usage, and rule auditability. However, pair selection, annotation instructions, and blinding are not described sufficiently. The manuscript should also consistently reflect the authors’ clarification that the collaboration score measures observable participation and citations, without establishing effective collaboration.

The examples include blocked orders, price-driven violations, and successful repairs of existing violations. These represent different behaviors, and it remains unclear how each affects the score. Excluding no-trade rounds from soft-rule scoring also needs explanation when inactivity is itself penalized.

Replay and reproducibility details remain incomplete. The July experiment is a useful addition, but its comparability with live evaluation depends on how historical news and other time-sensitive tool responses were reconstructed. Model settings, evaluation artifacts, and plans for comparing future runs also need clearer documentation.

Questions For Authors:

What do the reported standard deviations measure—variation across decision rounds or independent runs? How many independent runs underlie the April and July results?

How were the human-evaluation pairs selected, what instructions were provided, and was the annotator blinded to model identities and metric scores? Please also align the manuscript’s collaboration claims with the narrower interpretation given in the rebuttal.

When is rule compliance checked, and how are blocked orders, price-driven violations, successful repairs, and no-trade rounds treated?

How did the July replay restrict news and tool responses to information available at each historical decision time? Will the configurations, scoring code, and evaluation traces be released?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.