- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to evaluate LLM agents that run over long periods in changing real-world environments, using live financial markets as the test setting. The main contribution is LiveMACEBench. Five frontier LLMs paper-trade crypto and U.S. stocks for about 30 days under five matched setups: Base ReAct, Tool Use, Persistent Memory, Rule Following and Multi-Agent Collaboration. The authors score each setup with both returns and process metrics taken from full decision traces, using LLM judges and rule checkers. They report an "outcome-capability gap": return rankings change often over time, capability rankings stay stable, and similar returns can come from very different behavior.
Critical strengths
1- A clear and useful framing. The paper separates three things: having a mechanism, using it well, and benefiting from it (Sec. 1). The trace diagnostics support this. For example, Gemini stores 93 memories but has the lowest memory usage score (0.497), while GPT-5.4 retrieves memories 332 times but stores only one (Table 1).
2- Matched setups with full traces. All agents share one ReAct scaffold and the same live market, and every decision trace is recorded (Sec. 2). This lets readers locate specific failure modes, such as the concentration-limit errors in rule following (Sec. 3.2.3).
Critical weaknesses
1- The main "gap" result is an unfair comparison. Capability scores are running averages over many traces, which settle down by construction, while cumulative returns depend on the market path (Appx. C.7). So 0 vs 21 ranking reversals (Fig. 2a) is what you would expect statistically. It does not show that the capability metrics are more valid.
2- One run per setup, with no uncertainty. Each model-configuration pair runs once over one market window. Two runs failed and were dropped (Appx. C.3, C.5). Return and drawdown gaps of a few points are reported without error bars. For example, a +0.06 change in max drawdown is counted as a memory "improvement" (Table 1). The main tool-use judge is Qwen3.5, from the same family as one of the evaluated models (Fig. 3).
Questions for authors
1- (Clarification) Are capability rankings still more stable than return rankings if both use the same window? Examples would be daily capability scores against daily returns, or running-average returns.
2- (Clarification) Can you add repeated runs or a second market window for at least one capability, so that the setup effects have uncertainty estimates? 3- (Clarification) Did you validate the LLM judge scores against human ratings on a sample of traces? Does the Qwen judge give higher scores to Qwen3-Max?
Review
Summary: LiveMACE evaluates five LLMs in a 30-day live paper-trading study, examining tool use, persistent memory, rule following, and multi-agent collaboration. It combines trading outcomes with process diagnostics to show that profitability and observable agent behavior can give different assessments of the same model.
Strengths: Distinguishing access to a mechanism, effective use, and eventual benefit is valuable. The paper clearly motivates why trading returns alone cannot explain an agent’s strengths and limitations.
The traces reveal meaningful differences between retrieving and using memories, following rules and explaining them correctly, and consulting specialists and acting on their advice.
Five model families and four capability-focused configurations operate over persistent accounts during the same market period. This provides a useful foundation for comparative analysis.
Weaknesses:
The broader findings still lack independent replication. The July replay supports the tool-use ordering of two models across a second window. However, it does not establish the reliability of the reported return and risk differences across configurations. The results should therefore remain framed as observations about these configurations and evaluation periods.
Human agreement provides useful support for tool use, memory usage, and rule auditability. However, pair selection, annotation instructions, and blinding are not described sufficiently. The manuscript should also consistently reflect the authors’ clarification that the collaboration score measures observable participation and citations, without establishing effective collaboration.
The examples include blocked orders, price-driven violations, and successful repairs of existing violations. These represent different behaviors, and it remains unclear how each affects the score. Excluding no-trade rounds from soft-rule scoring also needs explanation when inactivity is itself penalized.
Replay and reproducibility details remain incomplete. The July experiment is a useful addition, but its comparability with live evaluation depends on how historical news and other time-sensitive tool responses were reconstructed. Model settings, evaluation artifacts, and plans for comparing future runs also need clearer documentation.
Questions For Authors:
What do the reported standard deviations measure—variation across decision rounds or independent runs? How many independent runs underlie the April and July results?
How were the human-evaluation pairs selected, what instructions were provided, and was the annotator blinded to model identities and metric scores? Please also align the manuscript’s collaboration claims with the narrower interpretation given in the rebuttal.
When is rule compliance checked, and how are blocked orders, price-driven violations, successful repairs, and no-trade rounds treated?
How did the July replay restrict news and tool responses to information available at each historical decision time? Will the configurations, scoring code, and evaluation traces be released?