acceptodds
Under review as a conference paper at ICLR 2027

Code‑Executable Judge: Mitigating Arithmetic Hallucinations for Wallet‑Agent Evaluation

Abstract

Large‑language‑model‑driven wallet agents often generate answers derived from thousands of billing records, requiring heavy numerical verification. Conventional LLM‑as‑Judge suffers severe arithmetic hallucinations, leading to inconsistent and unreliable evaluation results. To tackle this problem, we present Code‑Executable Judge: instead of letting LLMs directly compute numeric results for judgement, our approach dynamically generates Python verification code and validates answers via program execution. We conduct extensive experiments on wallet‑agent benchmarks. Compared with vanilla LLM‑as‑Judge baselines, our method achieves substantially higher agreement with human annotations. We further systematically analyze failure modes of code‑driven judging and discuss its practical limitations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Agent Evaluation, Executable Verification, Arithmetic Hallucination, Financial Agents, Tool-Augmented LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_N5BEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Code-Executable Judge, an LLM-based evaluation framework that generates Python verification programs to evaluate numerical claims over structured wallet-agent execution traces. Instead of relying on implicit arithmetic reasoning, the method reconstructs evidence, filters relevant records, aggregates values, and compares them through code execution, with a bounded probe–verify–reflect loop for self-correction. Experiments on 471 real-world wallet-agent interactions show substantial improvements over vanilla LLM-as-a-Judge under the same GLM-5 backbone.

Critical strengths
  1. The paper is grounded in a real deployment problem and highlights a practical limitation of LLM-based judging for data-intensive numerical tasks.
  2. The separation between reasoning and execution is well motivated. Using complete structured traces for executable verification helps avoid errors caused by truncated contexts and incorrect numerical reasoning.
Critical weaknesses
  1. The core idea of generating executable programs for judging is not novel. Prior work such as PAJAMA [1] and RewardAgent [2] has already explored similar program-generation-based judging approaches, but the paper does not cite these relevant works.
  2. The experimental evaluation is too limited to support the broad claims of the paper. The method is evaluated only on a self-constructed wallet-agent benchmark with 471 interactions, while the number of baselines is also limited.

[1] Huang, Tzu-Heng, Harit Vishwakarma, and Frederic Sala. "Time to impeach LLM-as-a-judge: Programs are the future of evaluation." arXiv preprint arXiv:2506.10403 (2025). [2] Peng, Hao, et al. "Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Questions for authors
  1. Did the vanilla LLM judge baseline receive the complete execution trace, or a condensed version of it?
  2. Can you provide ablations isolating the three main components?
1 comment

Thank you for recognizing the deployment motivation and the separation between reasoning context and execution evidence.

  1. Novelty. We agree that program-generation-based judging has prior work, and omitting PAJAMA and RewardAgent was a gap in our literature review. Our intended contribution is not the general idea of executable judging, but its operational design for large structured wallet traces: independently constructing evidence scope, retaining complete-record access outside the model context, and autonomously handling schema and program errors. We acknowledge that these design choices require a clearer comparison with existing methods and stronger component-level evidence.

  2. Experimental scope. We agree that the 471-instance wallet evaluation and limited baseline set do not establish cross-domain generality. On this dataset, the same-backbone GLM-5 comparison improves Bad Case F1 from 46.6% to 86.0%. Beyond this evaluation, CEJ has processed millions of real-world wallet-agent evaluation cases in production. We have cumulatively inspected nearly 10,000 production cases manually and observed judgment accuracy broadly consistent with that reported on the 471-instance dataset. This provides supplementary evidence of effectiveness at production scale under diverse real-world queries.

Reviewer_VwoYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a code-executable judge for wallet agents that explicitly generates Python verification code to validate results. To this end, code-executable judges are more robust than conventional LLM-as-a-Judge approaches throughout the wallet agent workflow. In addition, LLMs are known to suffer from hallucination. To tackle this challenge, this paper combines the verification process with a probe-verify-reflect loop. On 471 real-world wallet agent interactions, the proposed code-executable judge shows strong performance gains compared to conventional LLM-as-a-Judge.

Critical strengths
  1. This paper explores the LLM-as-a-Judge topic on Wallet Agent, which is an interesting application.
  2. The paper is relatively easy to follow.
Critical weaknesses
  1. The literature review is not comprehensive. LLM-as-a-Judge itself is a well-established field. The proposed methods seem relatively simple and may have been proposed in previous work.
  2. The exact prompt for the wallet agent and code-executable judge is not shared. Without this information, it is difficult to provide meaningful insight to the research community.
  3. The dataset selection and annotation details are missing. I would suggest performing the experiment on a similar publicly available dataset. Without such information, it is hard to evaluate the effectiveness of the proposed method.
Questions for authors

See weakness.

2 comments

Thank you for recognizing the practical relevance of wallet-agent evaluation. We agree that the submission needs stronger related-work coverage and reproducibility details.

  1. Contribution and prior work. We do not claim that generating code for verification is a new idea. Our focus is making numerical verification operate autonomously over large, heterogeneous industrial traces, including independent evidence selection, complete-record access, schema probing, and execution-feedback-based recovery. Subsequent deployment has processed millions of real-world wallet-judge cases.

  2. Prompt disclosure. To improve reproducibility, we provide the core numerical-verification prompt in a separate comment below. It specifies mandatory Python execution, independent evidence selection, prohibition of hard-coded operands, schema probing, tool interfaces, execution restrictions, and verification procedures. This is an excerpt from the production prompt; additional production-specific instructions are not disclosed.

  3. Dataset construction and annotation. Section 4.1 of the supplied revised manuscript describes proportional sampling from production traffic. The benchmark contains 471 interactions, with 368 Good and 103 Bad cases. Each instance includes the query, answer, and structured execution trace. Following an initial pool of 20 annotators, 10 selected annotators performed independent double annotation of every instance, with disagreements adjudicated by a senior reviewer. Annotators had access to complete traces. We note that off-the-shelf public benchmarks are poorly suited for the wallet evaluation scenario. Beyond the 471 benchmark set, we have additionally sampled and manually annotated nearly 10,000 instances from live production traffic after deployment.

Verifier Prompt Excerpt (Condensed English Translation)

The production prompt is in Chinese. This condensed translation preserves the core verification instructions. Additional production-specific instructions are omitted; this is not the complete prompt.


You are a numerical verification expert for wallet-agent answers. Verify supported numerical claims using Python execution.

1. Core rules

  • Never perform mental arithmetic. Every verification target must have an executed pyexec_tool call.
  • Independently select evidence from original records according to the claim’s scope before computing. Do not rely solely on records listed in the answer.
  • Never hard-code numerical operands copied from textual logs. The claimed value may be used for comparison.
  • Prefer positive selection of relevant records. Use exclusion only when supported by clear classification features.

2. Scope Verify checklist targets first, then up to three important additional numerical claims from the answer. Avoid duplicate checks. Verify numerical metric-card values.

Supported operations: sums, averages, counts, and billing ratios whose numerator and denominator can be independently derived.

For an individual transaction claim, locate the amount using stated time/payment restrictions and print matching details. Leave merchant/product association to the downstream judge.

Skip predictions, recommendations, illustrative data, nonnumerical statements, unrelated public facts, repayment amounts/schedules, investment-income figures dependent on settlement logic, and ratios not derivable from records. Principal, balances, and underlying spending remain eligible. Under the production policy, amounts/counts already stated in previous-turn answers are treated as trusted and skipped.

3. Tool interface Call pyexec_tool(code, target_desc):

  • get_node(node_id): retrieve complete structured node data. Copy the exact identifier from logs.
  • set_data(name, data) / get_data(name): save/retrieve intermediate evidence for reuse under identical filtering conditions.
  • print(): emit verification results.

target_desc must identify the target, node identifiers, and key filters. If revising an earlier conclusion, explain why.

4. Evidence and code

  • Inspect actual schemas; field names may be Chinese or English.
  • For outer containers, use defensive access with defaults. For essential record fields, detect and report missing values rather than silently substituting defaults.
  • Print the record count after retrieving a list.
  • Probe unfamiliar schemas using keys, value distributions, and at most five sample records.
  • Filter original transactions and recompute aggregates; do not substitute precomputed summary totals.
  • Coarse category fields may be unreliable. Use transaction-detail features where required by the evaluation policy. “Other” in a Top-N table represents the remaining records.
  • Include all relevant nodes. Print at most five matching records as evidence, including available title, merchant, category, time, and amount fields.
  • For each target, print its name, computed value, claimed value, and consistency result. Print division-derived values to three decimal places.
  • Supported libraries include json, math, re, datetime, collections, statistics, and pandas.
  • File access, network requests, process creation/termination, and multithreading/multiprocessing are prohibited.

5. Execution workflow

  1. Identify eligible checklist targets, scope restrictions, and data sources.
  2. Consolidate schema probing into one round; exceptionally large or multi-node tasks may require two.
  3. Preserve intermediate data for subsequent calculations where appropriate.
  4. Validate one simple verification call before parallelizing independent targets.
  5. Combine targets only when nodes and filters are identical and only the aggregation differs. Otherwise, use separate calls.
  6. Use execution feedback to correct schema, filtering, or program errors. Do not change evidence scope merely to match the answer.
  7. Summarize executed results in the final response. Tool outputs alone are not the final response; resolve errors and omitted targets with additional calls.
Reviewer_EA7oOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Code-Executable Judge, which writes python code to verify the traces of wallet agents. Over 471 wallet-agent traces, the method improves accuracy by over 30 points compared to the vanilla LLM-based judge.

Critical strengths
  1. The paper addresses a well-motivated problem.
  2. The data collection and annotation seem robust.
Critical weaknesses
  1. The paper lacks novelty. For instance, [1-6] augment LLMs with tools like code interpreters for verification purposes. It’s not clear what this paper adds beyond extending such techniques to the wallet domain.
  2. The calculator and code-executable experiments use different models, so it’s not possible to draw a meaningful comparison.
  3. It’s not clear what context the vanilla LLM-based judge has access to. For instance, if the judge can only see the truncated or condensed trace, this could explain some of the performance gap.
  4. Ablations over the methods component would be helpful e.g., condensation or the probe–verify–reflect loop.

[1] “Critic: Large language models can self-correct with tool-interactive critiquing.” International Conference on Learning Representations. 2024.

[2] “Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification.” International Conference on Learning Representations. 2024.

[3] “Tool-augmented reward modeling.” International Conference on Learning Representations. 2024.

[4] “Agent-as-a-judge: Evaluate agents with agents.” arXiv preprint arXiv:2410.10934. 2024.

[5] “Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems.” Association for Computational Linguistics. 2025.

[6] “Incentivizing agentic reasoning in llm judges via tool-integrated reinforcement learning.” International Conference on Learning Representations. 2026.

Questions for authors

See weaknesses.

1 comment

Thank you for recognizing the motivation of the problem and the data collection effort. We address your concerns below.

  1. Novelty and related work. We agree that using generated programs or external tools for verification is not new, and our related-work coverage was insufficient. Our intended contribution is the operational design for numerical judging over large, heterogeneous wallet-agent traces: independently reconstructing the relevant evidence, executing retrieval, filtering, aggregation, and comparison over complete structured records, and recovering from schema and program errors through a bounded probe–verify–reflect loop. We do not claim that code-based verification itself is novel. We acknowledge that the distinction from the cited approaches needs stronger positioning and empirical support.

  2. Different calculator and CEJ backbones. This is a valid concern. The calculator configuration was an earlier Qwen3-235B-based system, whereas the final CEJ uses GLM-5. Its results therefore constitute a historical system-level comparison, not a controlled estimate of the benefit of code execution. The same-backbone comparison in Table 1 is Vanilla versus CEJ using GLM-5: 61.0% versus 93.8% accuracy, and 46.6% versus 86.0% Bad Case F1. This comparison evaluates the complete CEJ framework, rather than isolating individual components.

  3. Baseline context. Vanilla receives the complete billing details, not a truncated or condensed version. However, long transaction lists make it harder for the model to reliably select relevant records and perform numerical reasoning; sufficiently large inputs can also exceed its context limit. CEJ addresses this by generating programs that retrieve and operate on the complete underlying records, while returning concise, structured verification results to the model. This preserves access to all evidence without placing every transaction in the model’s context. Executed code performs numerical aggregation reliably once the relevant records and operations are correctly specified, although evidence selection and program generation remain potential sources of error. The reported improvement therefore reflects the combined benefits of programmatic evidence access, compact reasoning context, and executable numerical verification.

  4. Ablations. We agree that controlled ablations are needed. The reductions in probing and execution rounds described in Section 4.4 are development observations, not controlled component ablations. We do not claim that they establish the independent contribution of condensation or reflection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.