- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Code-Executable Judge, an LLM-based evaluation framework that generates Python verification programs to evaluate numerical claims over structured wallet-agent execution traces. Instead of relying on implicit arithmetic reasoning, the method reconstructs evidence, filters relevant records, aggregates values, and compares them through code execution, with a bounded probe–verify–reflect loop for self-correction. Experiments on 471 real-world wallet-agent interactions show substantial improvements over vanilla LLM-as-a-Judge under the same GLM-5 backbone.
Critical strengths
- The paper is grounded in a real deployment problem and highlights a practical limitation of LLM-based judging for data-intensive numerical tasks.
- The separation between reasoning and execution is well motivated. Using complete structured traces for executable verification helps avoid errors caused by truncated contexts and incorrect numerical reasoning.
Critical weaknesses
- The core idea of generating executable programs for judging is not novel. Prior work such as PAJAMA [1] and RewardAgent [2] has already explored similar program-generation-based judging approaches, but the paper does not cite these relevant works.
- The experimental evaluation is too limited to support the broad claims of the paper. The method is evaluated only on a self-constructed wallet-agent benchmark with 471 interactions, while the number of baselines is also limited.
[1] Huang, Tzu-Heng, Harit Vishwakarma, and Frederic Sala. "Time to impeach LLM-as-a-judge: Programs are the future of evaluation." arXiv preprint arXiv:2506.10403 (2025). [2] Peng, Hao, et al. "Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
Questions for authors
- Did the vanilla LLM judge baseline receive the complete execution trace, or a condensed version of it?
- Can you provide ablations isolating the three main components?
Thank you for recognizing the deployment motivation and the separation between reasoning context and execution evidence.
Novelty. We agree that program-generation-based judging has prior work, and omitting PAJAMA and RewardAgent was a gap in our literature review. Our intended contribution is not the general idea of executable judging, but its operational design for large structured wallet traces: independently constructing evidence scope, retaining complete-record access outside the model context, and autonomously handling schema and program errors. We acknowledge that these design choices require a clearer comparison with existing methods and stronger component-level evidence.
Experimental scope. We agree that the 471-instance wallet evaluation and limited baseline set do not establish cross-domain generality. On this dataset, the same-backbone GLM-5 comparison improves Bad Case F1 from 46.6% to 86.0%. Beyond this evaluation, CEJ has processed millions of real-world wallet-agent evaluation cases in production. We have cumulatively inspected nearly 10,000 production cases manually and observed judgment accuracy broadly consistent with that reported on the 471-instance dataset. This provides supplementary evidence of effectiveness at production scale under diverse real-world queries.