acceptodds
Withdrawn from ICLR 2027

ProcGround: When Correct Outcomes Mask Ungrounded Decisions in LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Correct outcomes do not guarantee evidence-grounded decisions. When task priors and narrative cues agree with decisive evidence, agents can follow a plausible default and succeed without consulting that evidence. We therefore introduce **ProcGround**, a benchmark of 215 service tasks across 12 domains. Each task defines an Evidence–Decision Contract, which specifies the evidence required before the first consequential commitment, as well as the subsequent decision and execution requirements. Across 2,365 resolved trajectories from 11 models, Semantic Success is 46.2%, but Contract-Grounded Success is only 24.1%; 47.8% of semantically successful trajectories violate the contract. On valid minimal evidence-flip pairs, initially successful Evidence Bypass cases fail in the flipped world at a rate of 36.4%, compared with 12.5% for Contract-Grounded cases, linking the process distinction to behavioral vulnerability. The phenomenon also appears on external AppBench tasks: 58.5% of trajectories bypass decisive evidence, and 98.5% of these bypasses are masked by correct outcomes. In controlled interventions, a reminder checkpoint improves evidence seeking before commitment for two models, while revealing decisive evidence helps three models recover by the final answer. ProcGround complements outcome evaluation with observable grounding requirements and behavioral tests of sensitivity to decisive evidence.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Evaluation, Evidence Grounding, Process Evaluation, Counterfactual Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9oA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ProcGround, a benchmark for evaluating whether LLM agents make decisions based on the required evidence rather than simply producing correct final outcomes. It includes 215 tasks across 12 domains and uses Evidence–Decision Contracts and counterfactual evidence changes to evaluate agent behavior. Experiments across 11 models show a large gap between outcome correctness and evidence-grounded success, highlighting that agents can reach correct answers without following a reliable decision-making process.

Critical strengths

Meaningful improvement over outcome-only evaluation: The Evidence–Decision Contract framework provides a clear way to distinguish correct outcomes from decisions made without sufficient evidence. Evaluating evidence acquisition before consequential actions is particularly relevant for real-world agents where reaching the correct answer by chance is not enough.

Comprehensive experimental validation: The paper evaluates 11 models across 215 tasks and includes counterfactual evidence changes, external AppBench experiments, and controlled interventions. The human validation audits and detailed failure analyses further strengthen the findings and help explain why agents make unsupported decisions.

Critical weaknesses

Dependence on process annotations: The evaluation relies heavily on predefined evidence contracts and LLM-based judgments about whether agent trajectories satisfy them. Although the authors conduct human audits, agreement is lower for some important evaluation subsets, raising questions about the robustness of the reported grounding measurements.

Limited causal interpretation of counterfactual results: The counterfactual experiments use a selected subset of 59 tasks, and evidence-bypass and grounded trajectories are not randomly assigned. Differences in task difficulty or evidence accessibility could partly explain the higher failure rate among bypass cases, limiting how strongly the results establish a causal relationship.

Questions for authors

How sensitive are the reported grounding success rates to alternative reasonable definitions of required evidence and consequential commitment? Could the authors provide additional analysis showing whether these evaluation choices materially affect the model comparisons?

Could the authors further control for task difficulty and evidence accessibility when comparing counterfactual failures between evidence-bypass and contract-grounded trajectories? This would help determine how much of the observed difference is specifically associated with evidence bypass.

Reviewer_nrXAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether successful task outcomes can conceal ungrounded decisions in language model agents, and introduces ProcGround, a set of 215 tasks across 12 domains with explicit requirements for the evidence needing to be acquired by the agent prior to the first consequential action. The authors evaluate if agents acquire the required and authorized case-specific evidence prior to their first consequential action and whether it is correct, distinguishing these requirements from those that need to be met for the ultimate success of the task. Across 11 models, the authors find that 47.8% of successful task executions fail to meet the decision-process level requirements. Counterfactual experiments on selected cases modify decision-relevant facts and find higher failure rates amongst previously successful task executions that have failed to acquire the required evidence. Further analysis on AppBench examines the gap between successful outcomes and required evidence acquisition, and controlled experiments vary prompt wording and evidence presentation to investigate factors influencing agent behaviors. Audits by humans assess task specifications and evaluation labels. The paper’s contributions include an evaluation framework and benchmark that explicitly define and evaluate evidence requirements at consequential decision points, and an empirical investigation into how outcome-only evaluation can inflate the apparent reliability of agents' decision processes.

Critical strengths
  1. The paper makes process requirements explicit and testable, providing information beyond final task success. ProcGround specifies the required, authorized, instance-level evidence and verifies if agents do and do not acquire it before their first consequential action and whether that action is correct. Across 11 models, 47.8% of successful executions do not satisfy decision-process requirements, thus going beyond the observation that correct outcomes do not imply reliable processes and providing an operationalization and a demonstration of its prevalence, a valuable contribution to agent evaluation.
  2. Complementary validation connects the identified process failures to behavioral fragility: counterfactual tests on selected cases change decision-relevant facts and observe higher failure rates in past successful executions lacking the required evidence. Additional analyses on AppBench uncover a similar gap between successful outcomes and evidence acquisition in a different task environment, and human audits evaluate task specifications and evaluation labels, building confidence in the main results and providing evidence that the observed evidence-acquisition failures are associated with behavioral fragility under changes to decision-relevant facts, supporting the practical value of complementing outcome evaluations with decision-process analyses.
Critical weaknesses
  1. The sensitivity of secondary findings to annotation uncertainty remains unclear. The paper contains extensive human audits, including a task-contract census and corrections that maintain the main outcome–process gap. Frozen–human agreement is lower for Original-side Evidence Bypass in Final-59 (85.0%) and Semantic Success in AppBench (82.1%). Those labels directly affect counterfactual group assignment and the external outcome-masking analysis. The paper does not establish the sensitivity of those secondary findings to the observed labeling disagreements. This weakens confidence in the magnitude of those secondary results, but not the main audited finding.
  2. Selective counterfactual construction limits the generalizability of the process metric’s association with behavioral fragility. Final-59 is selected around previously successful Evidence Bypass executions rather than representing the entire task set. The per-model comparisons, equal-model-weight analysis, and held-out receiver results provide useful evidence that the observed association is not an artifact of pooling models. Despite that, the evidence-missing and contract-grounded groups may differ in task difficulty or evidence accessibility, and some grounded comparison groups are small. The results therefore support an association within the selected cases, but more limited evidence about the consistency with which the process metric identifies fragile executions across workloads.
Questions for authors
  1. Sensitivity to annotation disagreements (clarification and additional analysis using existing records): Given the reported frozen–human agreement for Original-side Evidence Bypass in Final-59 and Semantic Success in AppBench, do the associated findings hold under the application of human-adjudicated labels to the audited records? Might the authors report the direction of the relevant label changes and their effect on the counterfactual group assignment or the outcome-masking estimate, distinguishing audited-subset results from any full-cohort sensitivity analysis available? This would help establish my confidence in the robustness of the secondary findings.
  2. Task-level comparability in the counterfactual analysis (clarification and additional analysis using existing results): Table 2 and the equal-model-weight analysis address the issue of model composition. Might the existing data also allow a form of analysis that accounts for potential differences in task composition between the Evidence Bypass and contract-grounded groups, for example by comparing on tasks represented in both groups? If so, does the failure-rate association continue to hold, and how much of the selected sample is relevant to that comparison? This would help establish whether the observed association reflects the fragility of the process involved beyond the possible differences in task difficulty level or accessibility of evidence.
Reviewer_1J3AOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies cases where an agent completes a task correctly without obtaining the evidence needed for its decision. The authors construct 215 service tasks, evaluate outcomes and decision processes separately, and change decisive evidence to test whether agents adjust their actions. Across 11 models, final success is 46.2%, while success that also satisfies the process requirements is 24.1%. The paper examines the same phenomenon on AppBench and tests interventions involving additional instructions and evidence provided in advance.

Critical strengths

For tasks that depend on current state and user authorization, final outcomes alone are insufficient. Separating verification before commitment from recovery after an error highlights problems that conventional success rates can miss.

Critical weaknesses
  1. I am not fully convinced by the rationale for the process metric. A lack of recorded external verification does not necessarily mean that the model lacks sufficient knowledge. Checking current state is necessary, but this does not justify imposing the same verification requirements on every decision. The first-commitment boundary also seems overly strict in some cases. In the appendix’s airport example, an update attempt is rejected without changing any state; the agent then checks the evidence and completes the task, yet receives a process-failure label. For interfaces that permit safe trial and error, this behavior may be reasonable. The paper needs to connect these judgments more directly to actual error risk. GroundEval[1] already distinguishes answer correctness from evidence-path validity, so a direct comparison would also help establish what the first-commitment boundary adds.

  2. The experiments do not sufficiently rule out alternative explanations. The 59 counterfactual tasks are selected from cases where three models originally succeeded while bypassing evidence. The grounded and bypass groups may also differ in task difficulty, so their failure-rate gap need not primarily reflect verification behavior. The grounded groups for Sonnet and Qwen contain only five and seven trajectories, respectively. Furthermore, evidence provided in advance appears in a user message, whereas evidence obtained through retrieval arrives as a tool response. This changes both availability and source. A model that declines to trust data in a user message may be assessing source reliability, rather than failing to use evidence it has already obtained.

[1] GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

Questions for authors
  1. How would the main conclusions and model rankings change if safe attempts that leave the state unchanged were scored separately from actions with actual consequences? Which tasks require current external information, and which could plausibly be completed using existing knowledge?

  2. Does the counterfactual failure-rate gap persist when tasks are selected independently of original Evidence Bypass behavior and model and task differences are controlled?

  3. If the evidence’s message role and source are held constant, with only its initial availability changed, do the conclusions about evidence use and later recovery still hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.