acceptodds
Withdrawn from ICLR 2027

Separating Structural Semantics from Inference Allocation in LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-agent comparisons often change both what the model is asked to do and how much inference it receives, then treat realized resource consumption as a control. We separate structural semantics , ex-ante allocation and realized downstream cost , and explain why adjusting for does not in general estimate the effect at fixed allocation. Our intervention requests separately seeded candidates under byte-identical prompts; in sequential control a fixed plurality rule selects the action. We audit prompt invariance using full-message hashes in program synthesis and system-prompt hashes plus prompt-construction tests in control. In a preregistered factorial over four models, four cooperative-cooking layouts and five seeds, allocation contrasts range from to soups per episode. All four percentile-bootstrap intervals lie below zero, but each admits a drop smaller than our preregistered practical margin, so these data do not establish practical harm. Structural effects remain imprecisely estimated. In a separate matrix of 2,624 HumanEval candidate pools, both the frozen and corrected original-test scorers place the structural effect at within the fixed equivalence margin. This oracle-success metric has no aggregator and is not a replication of the control result. Observed changes run opposite to both preregistered mechanism predictions; the mechanism remains unresolved. The results concern the tested sampling-and-plurality policy, whose candidate count and selection rule are not separately identified.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, inference-time compute, test-time scaling, self-consistency, plurality voting, preregistered factorial, causal confounding, agent scaffolds

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PChvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly contributes evaluation methods and experimental practices for LLM multi-agent systems. It distinguishes three factors: structural prompts, the number of candidates and their selection rule specified in advance, and actual resource consumption. The authors propose an auditable criterion for checking that requests remain unchanged when generating 1, 3, or 5 candidates. The cooperative cooking experiment uses a fixed voting rule to select actions, while HumanEval measures whether a candidate pool contains a correct program, without voting.

Critical strengths
  1. The paper has a clear research question: Fixing the number of candidates and matching actual token consumption answer different questions. Selecting runs based on costs produced after treatment can change what is being estimated.

  2. The paper is honest and cautious interpretation: The authors do not interpret non-significant structural effects as “no effect,” or generalize the decline under voting to “more inference is always harmful.” Scoring corrections and their impact are also disclosed separately.

Critical weaknesses
  1. Repeated sampling, degradation under voting, and the risks of adjusting for post-treatment variables have already been studied. The main additions are an auditable criterion, preregistered factorial experiments, and fixed margins for practical importance. The paper needs to explain more clearly what new insights this combination provides beyond existing evaluations.

  2. The voting experiment tests only one fixed rule, without comparing other candidate-selection methods. The negative results therefore apply only to the combined sampling-and-voting policy. The observed changes also go against both preregistered mechanism predictions, leaving the reason for the performance decline unresolved.

  3. After scoring corrections, the single-candidate pass rate on HumanEval reaches 92.7%, leaving limited room for improvement. These results therefore cannot establish that structural prompts are unhelpful on harder tasks, nor should they be treated as a replication of the cooking results.

Questions for authors

See weakness

Reviewer_qvy9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ABSTRACT

"LLM-agent comparisons " -> please provide the context and explain llm-agent and comparsions what is the model here? what is the inference here? what is realized resource consumption? what is a control?

"We separate structural semantics ..." -> none of these terms are well defined; there is no motivation for such separations; in particular, what is allocation here? what does ex-ante mean? wnat downstream? what is the meaning of realized vs non realized? What is the meaning of C and its adjustment?

"Our intervention requests " -> how do your previous sentences lead to this? what is intervention here? what is request? what is m? why m has to be in these ranges? what are byte-identical prompts?

I can't understand any of the sentences here; they are not readable for humans

1 INTRODUCTION

"Suppose a role prompt ..." -> what is scores here? what are these things? role prompt? a critic? They are not comparable properties

"Such a comparison " -> what comparison? there isn't one right now

Critical strengths

I can't understand any of the sentences in the paper. They are not readable to humans.

Critical weaknesses

I can't understand any of the sentences in the paper. They are not readable to humans.

Questions for authors

I can't understand any of the sentences in the paper. They are not readable to humans.

Reviewer_bm6qOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Comparisons of LLM agents often change both what a model is asked and how much inference it gets, then "control" for the tokens spent. This paper argues that three things should be kept apart: the semantics of the prompt, the allocation fixed in advance, and the cost actually incurred. Adjusting for realised cost, a post-treatment variable, does not recover the effect of allocation. The intervention is clean in principle: draw m∈{1,3,5}m \in \lbrace 1, 3, 5 \rbrace candidates under byte-identical prompts and execute the plurality action. In a preregistered factorial on a two-agent cooking task, more candidates consistently lower reward, but by amounts within the preregistered margin. The effect of a structural prompt is unresolved, a code task shows no difference, and two proposed mechanisms do not fit.

Where it sits (October 2026): the case that agent benchmarks must report cost is established (Kapoor et al., "AI Agents That Matter", and FEval-TTC), non-monotone returns to voting are known (Chen et al., 2024, cited), and 2026 work shows that the value of majority voting depends on the candidate supply and the selection rule. The post-treatment argument is standard causal inference, which the paper cites. What is new is the crossed design with practical margins, and that design needs a strong empirical result to carry the paper.

Critical strengths
  1. Reproducibility is exemplary. A single script regenerates every reported number from frozen logs, and the authors disclose their own defects. That is what made checking possible, and the community should copy the practice.

  2. The crossed S×mS \times m design of semantics against allocation, judged against fixed practical margins and with "unresolved" as an allowed verdict, is a sound template, and declining to call the harm practically established is the right call.

Critical weaknesses
  1. The paper's methodological claim, that its mechanism tests were preregistered, is not supported by the supplied preregistration. The mechanism section is presented as preregistered, but the preregistration files contain no mechanism hypothesis, and the analysis code frames the second hypothesis as the explanation that survived after the first failed. Several preregistered estimands and sensitivities are also not reported, and one omitted sensitivity, the cell bootstrap, disagrees with the primary verdict on the structural prompt. For a paper whose contribution is mainly protocol discipline, this is the central issue.

Recommendation: relabel the section as exploratory and report every registered analysis.

  1. The allocation effect looks more like a property of the harness than of inference allocation. Plurality over samples picks the most common action, which lowers randomness. In this executor, deterministic policies score nothing and many episodes score zero, and the logs show agents repeating the same goal more often as the number of samples grows, closely tracking the drop in reward. No alternative selector is tested, so allocation cannot be separated from plurality voting, and with four models no exact test can reach conventional significance.

Recommendation: an entropy-matched control, either a single sample at lower temperature or a uniform selector over valid samples (already in the supplement), would decide the mechanism.

Questions for authors

Questions that could change my assessment:

  1. Is there a timestamped, pre-collection document stating both mechanism hypotheses? If not, will you retitle Section 6 as exploratory and correct the abstract and introduction?
  2. Does an entropy-matched control reproduce the drop in reward?
  3. Can you report every preregistered estimand and sensitivity, including the cell bootstrap, together with a model-level exact or randomisation test?

Clarifications and recommendations (not affecting my score):

  1. Why does the Domain 1 prompt hash use a template recomputed at the end of the episode rather than the messages actually sent?
  2. The Lean development checks algebraic identities and takes the probabilistic step as an assumption, so the text should not describe the concentration bound as formalised.
  3. Please cite LLM-Coordination (Agashe et al.), whose code the supplement redistributes, as well as Kapoor et al. and FEval-TTC.
  4. A harder code benchmark than HumanEval would give the second domain room to show an effect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.