acceptodds
Under review as a conference paper at ICLR 2027

An Estimand-Based Evaluation Framework for Compositional Reasoning Pipelines

10 views

Abstract

Stage-level scores for compositional reasoning depend on which examples are scored, what upstream information is available, and which chance reference is used for comparison. These are distinct choices: scoring a stage only on examples whose upstream stages were answered correctly selects a model-specific population and changes the reference against which its score should be read, whereas supplying ground-truth upstream answers changes what the model receives while retaining all applicable examples. We present an estimand-based framework that makes these choices explicit. It distinguishes natural, prefix-conditional, and oracle-conditioned accuracy, pairing each with a chance reference matched to both its population and its upstream information. It additionally measures a signed selection gap by scoring the same oracle predictions on different populations. In a controlled shape-grammar study with five multimodal models, final-stage conditional accuracy reaches 98.45–99.67% but covers only 6.2–15.1% of applicable examples and ranks models differently from full-population accuracy. Holding oracle predictions fixed for two models, restricting evaluation to natural survivors raises accuracy by 7.15 and 12.97 percentage points at the rule stage and by 7.07 and 11.30 points at the final stage. Final-stage survivors reach 93.37–96.49% accuracy under fixed oracle predictions, still below their natural prefix-conditional scores. Furthermore, survivor-matched chance references deviate from full-cohort baselines. In a reanalysis of published results, accounting for coverage changes the preferred retrieval context size in five of eight comparisons. These results suggest the necessity of reporting population, input regime, coverage, and a matched chance reference alongside every stage score.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
reasoning pipeline evaluation, compositional reasoning, evaluation estimands, selection bias, chance baselines

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CjafOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to fairly evaluate models on tasks that involve multiple reasoning stages. The authors point out that evaluating a later stage only on examples where all earlier stages were correct can be misleading, because different models may be evaluated on different and much smaller subsets of examples.

To address this issue, the paper proposes a framework that compares three types of accuracy: accuracy on all examples, accuracy on examples with correct earlier stages, and accuracy when the model is given correct answers for earlier stages. It also analyzes how selecting examples affects accuracy and the appropriate random-guessing baseline.

The authors evaluate five multimodal models on 4,100 synthetic visual reasoning tasks. They find that later-stage accuracy can reach nearly 99% when only earlier-stage successes are considered, even though these examples represent just 6–15% of the applicable dataset. They also show that different evaluation methods can change model rankings.

Critical strengths

Strenghts

  1. The paper discusses an important issue in multi-stage reasoning evaluation, showing how conditional accuracy can be misleading due to reduced coverage and model-specific sample selection.

  2. The proposed framework is well-structured, with clear mathematically definitions of different accuracy metrics, chance baselines, and selection gaps.

  3. The experiments demonstrate how prefix conditioning affects coverage, accuracy, and model rankings. The additional analysis of FinRAG-QA further supports the practical relevance of the proposed framework.

Critical weaknesses

Weakness

  1. The reanalysis of FinRAG-QA is a valuable addition that demonstrates the practical relevance of the proposed framework. However, the external validation is limited to a single benchmark, while the main experiments rely on a synthetic shape-grammar dataset, where the six stages are evaluated from a single JSON output rather than an interactive reasoning process. I encourage the authors to consider one or two additional benchmarks, such as WebArena or ALFWorld (or other suitable benchmarks of their choice), to further investigate coverage and selection gaps in realistic agent tasks. This would strengthen the evidence for the framework's generalizability.

  2. The chance baselines are straightforward to compute in the synthetic setting, where the answer spaces and guessing distributions are predefined. However, their applicability to open-ended tasks remains unclear. For example, in software development tasks, how should one define the probability of an agent randomly generating a correct solution? Since the space of valid solutions is difficult to enumerate, it is unclear how the proposed chance baselines would generalize to such settings. Further discussion of this limitation or potential approaches to handling open-ended tasks would strengthen the paper.

Questions for authors

See weaknesses above. If the authors can adequately address the above two weaknesses, I would be inclined to support acceptance.

3 comments
AuthorsOpenReview ↗

ALFWorld Evaluation and Chance References for Open-Ended Tasks

Dear Reviewer,

Thank you very much for your insightful comments that helped us to substantially improve the manuscript.

1. New ALFWorld pipeline

Following your suggestion, we added ALFWorld as a second external benchmark alongside FinRAG-QA. Section 5.5 and Table 3 report coverage and fixed-oracle selection gaps in an interactive agent task, with the full protocol in Appendix I.

We evaluate GPT-5.6 Terra and Kimi K3 on all 134 unseen text games using four milestones: discovery, acquisition, processing, and terminal success. Actions are executed sequentially and change the environment encountered downstream. For each later stage, an expert prefix completes the preceding applicable milestones before handing control to the model. We retain natural-run survivor membership and score the same oracle outcomes across survivors, non-survivors, and all applicable games.

This also extends the external evidence beyond coverage accounting: we measure fixed-oracle selection gaps, natural–oracle differences, and executable reference-policy success. Both models perform near ceiling. Across both models and all four milestones, survivor coverage ranged from 92% to 100%. Fixed-oracle selection-gap estimates ranged from −0.35 to 0.00 percentage points. We present these as small, unresolved contrasts, rather than evidence that selection is generally absent. The framework measures population contrasts without requiring them to be large or positive.

Executable references further distinguish evaluation regimes: terminal random-policy success is 0.33% from initial states and 11.81% from expert checkpoints, while natural survivor checkpoints yield different references. We explicitly distinguish these state/history/budget regimes.

2. Defining chance references for open-ended tasks

We agree that defining chance baselines in open-ended tasks is more challenging than in our controlled synthetic setting, particularly because the space of valid solutions cannot generally be enumerated.

To address this concern, we have generalized the definition of chance success in Section 3.2 from the probability of guessing a designated ground-truth answer to the probability that a declared reference policy produces an accepted solution. Specifically, we define the reference success probability as:

bids=EY∼qids[Aid(Y)]b_{id}^{s} = \mathbb{E}_{Y \sim q_{id}^{s}}[\mathcal{A}_{id}(Y)].

Here, qidsq_{id}^{s} is the output distribution of the declared reference policy under information regime ss, and Aid(Y)\mathcal{A}_{id}(Y) indicates whether the task accepts output YY.The original finite-answer guessing baseline is a special case of this formulation. Importantly, the information- and population-change decomposition in Equations (2)–(3) remains valid without requiring enumeration of the solution space.

We have also made the following revisions:

  1. Appendix D.2: We added a software-engineering example. For software repair, we specify a patch-generation policy, its permitted inputs and budget, and an acceptance criterion such as passing the prescribed tests. We estimate reference success by sampling patches and reporting the fraction accepted. This requires no enumeration of valid patches.

  2. Related sections: We revised the Abstract, Introduction, Limitation, and Conclusion to reflect the generalized formulation.

We hope the new experiment and the explicit reference-policy definition address both concerns, and we appreciate your willingness to reconsider the paper in light of these revisions.

Thank you for the response. I am inclined to maintain my original score.

Thank you for considering our response and the revised manuscript. If you have a remaining concern about either the ALFWorld evaluation or the reference policy for open-ended tasks, we would appreciate knowing which aspect needs further clarification.

Reviewer_fLZfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a stage-level score in a compositional pipeline can only be interpreted once 3 choices are fixed:

  • scored population: all applicable examples, or only "survivors" whose upstream stages were correct.
  • input regime: the model's own upstream outputs, or ground truth supplied as input.
  • chance reference. The paper defines 3 accuracies: unconditioned (natural), prefix-conditional, and oracle-conditioned. It also defines 3 uniform guessing chance references. Their differences split into an information-change term and a population-change term. Survivor coverage is written as a product of conditional pass rates. A signed selection gap is obtained by scoring the same oracle-run predictions on natural survivors, natural non-survivors, and all applicable examples. The authors note that these identities are standard. The stated contribution is to make the choices explicit and to show their consequences. The validation in the paper has 4 parts:
  • synthetic study. A procedurally generated shape-grammar visual task with 4,100 puzzles.
  • fixed-oracle selection gaps.
  • matched chance references.
  • external reanalysis. The paper recommends that every stage score should be reported with 4 fields: the applicable count, the evaluated count and its population, the accuracy, and a chance reference computed on that same population.
Critical strengths
  1. The paper proposes a reusable way to separate who is scored from what is predicted. The paper holds one oracle run fixed and rescores it on natural survivors, non-survivors, and all examples. This isolates a population contrast with the predictions held constant.
  2. The warning about near-ceiling conditional stage scores is valid and well-quantified. A final-stage conditional accuracy of 98.45 to 99.67% describes only 129 to 314 of 2080 applicable puzzles. All five 95% CIs share a common interval, whereas unconditioned accuracy spans 11%. Staged diagnostic benchmarks often report accuracy given correct upstream answers, so this is a useful cautionary result. It is also the strongest support for the 4-field reporting recommendation.
Critical weaknesses
  1. The evidence that current practice misleads is narrow. e.g. The testbed is not a pipeline. It is one synthetic task whose "stages" are a post-hoc partition of a single JSON output. The fixed-oracle evidence is narrow. Selection gaps are shown for only two models at two stages. The single external case tests only coverage (J = p·a). In 3 of its 5 "reversals", the conditional-accuracy preference rests on differences of 0.3 to 1.1 %, which is within sampling error.

  2. Matched chance references have no demonstrated consequence. In Table 9, the population term is at most 1.1%, and it is 0 at 3 stages. Using the "wrong" reference changes chance-corrected accuracy by at most 0.6%. Only lift moves, and lift is degenerate when chance is near 0. The convention is also applied inconsistently: the released analyzer uses 40 settings for non-uniform scaling, whereas Table 8 states 50.

  3. The final-stage "oracle" is not the estimand defined in Eq. 1. Table 1B uses the MS run, which supplies only mode and shapes. The survivors' 93.37 to 96.49% being "still below" their natural conditional scores is therefore expected whenever upstream and operand answers are coupled within a response. Table 5 shows this coupling is substantial: 99.2%/99.5% on the MS run's own survivors versus 93.37%/96.49% on natural survivors. The framework does not attribute the gap that this coupling creates.

Questions for authors
  1. Did you run a complete-prefix oracle for the operand stage, supplying mode, shapes, rules, arguments, and algebra as Eq. 1 requires? Do the Table 1B gaps hold under it? The MS run also yields oracle predictions for arguments and algebra. What are the fixed-oracle selection gaps at those two stages, and is any of them negative? A negative gap would directly support "signed selection" under fixed predictions.
  2. Which answer space for non-uniform scaling produced Tables 2 and 9: 50 settings (Table 8) or 40 (released analyzer)? Does the mode-stage reference account for the structural-consistency requirement used in mode scoring?
  3. Is there any case, in your data or in a published benchmark, where a survivor-matched rather than full-cohort reference changes a conclusion (a ranking, a significance call, or an above-chance claim) on a metric other than lift? If not, why should a matched chance reference be one of the four required reporting fields?
2 comments
AuthorsOpenReview ↗

Pipeline Validation, Matched References, and the Operand Oracle

Dear Reviewer,

Thank you for your insightful feedback. Our response is as follow.

W1. Evidence beyond the shape grammar task

We agree that the original empirical evidence was limited and have expanded the evaluation while narrowing the interpretation.

Section 5.5 now reports a four-stage ALFWorld study using GPT-5.6 Terra and Kimi K3. Each action changes the environment encountered downstream. For each later milestone, we replay an expert prefix, hand control to the model, and score the resulting oracle outcomes on survivors, non-survivors, and all applicable games. Coverage ranged from 92% to 100%, and fixed-oracle selection gaps ranged from −0.35 to 0.00 percentage points. Terra’s acquisition result illustrates what the framework can distinguish even at full coverage. All 134 games survived discovery, so the selection gap was exactly zero. On those same games, Terra acquired the target in 128 natural runs and all 134 expert-prefix runs, giving a natural-minus-oracle contrast of −4.48 percentage points.

We also qualified the FinRAG-QA result in Appendix G. In five of eight retrieval–generator combinations, conditional accuracy and joint success favor different context sizes by point estimate. We use this example to show how the choice of objective affects the reported comparison.

W2 & Q3: When a matched chance reference matters

We agree that the effect of chance correction should be distinguished from the additional effect of matching the reference population. Across our five-model analysis, survivor-matched CCA changes score magnitudes but preserves the conditional point-estimate rankings.

Nevertheless, population matching serves a precise measurement purpose. For survivor accuracy aSa_S, the comparison decomposes as follows:

aS−Ball+=(aS−BS+)+(BS+−Ball+).a_S - B_{\mathrm{all}}^{+} = (a_S - B_S^{+}) + (B_S^{+} - B_{\mathrm{all}}^{+}).

Comparing against the full-cohort reference therefore combines performance above the survivor-matched reference with a difference in reference difficulty between populations. Matching removes this discrepancy, whose numerical importance is an empirical question, rather than something the framework assumes.

We have therefore revised the reporting recommendation: population- and information-matched references are required when making reference-relative performance claims, rather than being prerequisites for reporting raw accuracy. We retain the recommendation to declare a reference when interpreting performance beyond a baseline, without claiming that matching must change a ranking or significance decision. Because lift is highly policy-dependent when its denominator is small, we emphasize CCA and omit those lift point estimates from Table 2.

W3 & Q1: Complete-prefix operand oracle and signed selection

We agree that the original MS operand audit did not instantiate the complete-prefix operand estimand in Eq. 1. We have performed a complete-prefix operand intervention for GPT-5.6 Terra and Kimi K3 on all 2,080 applicable examples, supplying ground-truth mode, shapes, rules, arguments, and algebra. The selection gaps persist (revised Table 1B): Terra’s survivor and full-population accuracies are 95.59% and 90.96%, giving a 4.63 point gap; Kimi’s are 98.95% and 90.53%, giving an 8.42 point gap.

Your observation about within-response coupling also prompted us to revise the interpretation. We removed the statement that oracle survivor accuracy remains below natural conditional accuracy for both models. The natural–oracle difference on the same survivors is now 3.43 points for Terra and zero for Kimi.

Appendix B, Table 7 reports the requested comparisons using the existing MS predictions. At arguments, selection gaps are +17.28 points for Terra and +15.74 for Kimi. At algebra, they are +0.64 and −5.30 points, respectively. Kimi’s Algebra result provides direct evidence of negative selection with predictions and input regime held fixed.

Q2: Which answer space and Mode reference were used

Our results use M=40 nonuniform-scaling settings as in the code. We have corrected Table 10 (previously Table 8). We also verified that the Mode-stage reference remains 1/31/3 under the implemented scoring and reference procedure, including its treatment of structural consistency.

Thank you again for the careful reading and constructive feedback.

Thanks for the response, although the response is carefully sufficient, I believe I will maintain my current rating

Reviewer_bcJeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how stage-level evaluation changes with the population being scored and the upstream information supplied to a model. It distinguishes natural full-population accuracy, accuracy conditional on correct upstream predictions, and accuracy under supplied ground-truth upstream inputs. It also proposes matching chance references to both the evaluated population and the available information.

The empirical study uses a synthetic shape-grammar task evaluated across five multimodal models. Conditional downstream accuracy approaches 100% while covering only a small fraction of applicable examples. For two models, the paper holds oracle predictions fixed and compares accuracy on natural survivors, non-survivors, and the full population, demonstrating substantial selection gaps. A reanalysis of FinRAG-QA further shows that accounting for retrieval coverage can change the preferred configuration under a joint-success objective.

The measurement question is relevant, and the central selection results are supported. However, the evidence for the broader framework remains limited by the post-hoc nature of the stages, incomplete final-stage oracle inputs, and insufficient detail concerning chance distributions and equivalent programs. My overall recommendation is borderline reject.

Critical strengths
  • A clear measurement question supported by a well-designed population comparison. The paper correctly distinguishes changing the evaluated population from changing the information supplied to a model. Its strongest experiment scores the same oracle predictions on natural survivors and non-survivors. This directly establishes that restricting the population changes reported accuracy under a fixed prediction regime. The reported selection gaps of approximately 7–13 percentage points, with intervals excluding zero, are meaningful evidence that the distinction matters in this testbed.

  • Careful interpretation and transparent reporting.

The manuscript reports applicable and surviving counts, separates conditional from full-population accuracy, and acknowledges that the probability identities are established rather than novel theoretical results. It also avoids interpreting oracle accuracy as natural accuracy with selection bias removed. The FinRAG-QA reanalysis provides a useful practical illustration while explicitly distinguishing joint gate-and-answer success from unrestricted answer accuracy. These qualifications strengthen the credibility of the narrower conclusions.

Critical weaknesses

The empirical setting does not sufficiently validate the framework for compositional reasoning pipelines. Section 4.1 states that each model generates one structured JSON response in a single completion. The six stages are evaluated afterward and do not correspond to separately executed reasoning steps. This is a valid setting for studying conditional scoring, but it does not establish how the framework behaves in a pipeline where upstream predictions actually become downstream inputs. Associations between fields may partly reflect within-response consistency rather than sequential error propagation.

The operand-stage audit also supplies only mode and shapes, rather than the complete upstream prefix of rules, arguments, and algebra. The paper acknowledges this partial-oracle setting. The resulting selection comparison remains valid under that input regime, but does not isolate operand performance after upstream uncertainty has been removed.

The external reanalysis validates coverage accounting only; it does not test the fixed-oracle selection or matched chance-reference components outside the synthetic task. Since the underlying identities and evaluation concepts are established, stronger empirical validation is important to establish the significance of the integrated framework. An actual multi-stage application and a complete-prefix operand intervention would materially strengthen the contribution.

The scoring and chance-reference assumptions need more precise specification and robustness checks. Appendix D lists answer domains and parameter counts, but these do not uniquely determine a guessing distribution over complete structured programs. Uniform sampling over complete programs can differ from uniform sampling over successive field choices, particularly when the number of transformed objects and rule applications varies. The manuscript should specify the distributions sufficiently precisely to reproduce the aggregate chance references and assess sensitivity to reasonable alternatives. This is particularly important for lift values with very small chance denominators.

The task also requires clearer treatment of equivalent or non-identifiable programs. The declared grammar allows nonuniform scaling with equal axis factors, which can coincide with uniform scaling. More generally, setting the number of demonstrations to one plus the number of parameters does not itself guarantee program identifiability. The PDF does not explain whether equivalent programs are accepted or how ambiguous examples are excluded. This affects not only accuracy but also the upstream correctness labels that define survivor membership.

Finally, the reported intervals do not establish robustness to repeated model generations or alternative defensible stage orderings. The conditional model rankings have strongly overlapping intervals, so their point-estimate reordering should remain a descriptive observation rather than evidence of a stable reversal in model capability. These concerns do not invalidate the observed fixed-prediction selection gaps, but limit how broadly they can be interpreted.

Questions for authors
  • How are semantically equivalent programs scored, and how is identifiability from the demonstrations checked? In particular, how are uniform scaling and equal-factor nonuniform scaling handled? Can you quantify whether equivalence handling changes survivor membership or the reported selection gaps?
  • What are the exact guessing distributions used to obtain the chance references? Are predictions sampled uniformly over complete programs or through a hierarchical procedure over fields? How sensitive are the main chance-normalized conclusions to reasonable alternative distributions?
  • Are complete-prefix oracle results available for the operand stage? The current mode-and-shapes intervention supports a fixed-regime population comparison, but leaves other upstream predictions uncontrolled.
  • Do the selection gaps and survivor memberships remain stable across repeated generations? What population and sources of randomness are the reported confidence intervals intended to represent?
  • How sensitive are the results to alternative stage orderings consistent with the task’s dependencies? Since the stages are applied after a single completion, this would help distinguish properties of the task from properties of the chosen scoring protocol.
2 comments
AuthorsOpenReview ↗

Pipeline Validation and Robustness Checks

Dear Reviewer:

Thank you for your constructive insights. We address your concerns below.

1. New agent pipeline and external validation

We added an ALFWorld study that addresses both evaluation in an executed pipeline and external validation (Section 5.5, Appendix I). We evaluate GPT-5.6 Terra and Kimi K3 using four milestones. Each action changes the environment encountered downstream. For each later milestone, we replay an expert prefix and hand control to the model. We then compare the same oracle outcomes across groups defined by the natural run. Across both models and all four milestones, survivor coverage ranged from 92% to 100%, and selection-gap estimates ranged from −0.35 to 0.00 percentage points.

2. Complete-prefix operand oracle

We have performed a complete-prefix operand oracle for GPT-5.6 Terra and Kimi K3, supplying ground-truth mode, shapes, rules, arguments, and algebra. The selection gaps persist under this intervention (revised Table 1B). Terra’s oracle accuracy was 95.59% on natural survivors and 90.96% on all applicable examples, a gap of 4.63 percentage points. Kimi’s corresponding accuracies were 98.95% and 90.53%, a gap of 8.42 points.

3. Reference-policy specification and sensitivity

Appendix D.1 now specifies what the reference policy receives, how it samples identities and legal arguments, which outputs the scorer accepts, and how we average item-level probabilities. Appendix D.2 and Table 12 compare hierarchical field sampling (H) with uniform sampling over complete syntactic rule programs for the retained transformed-shape set (C). On the matched Terra cohort, survivor Rules chance-corrected accuracy (CCA) was 64.04% under H and 63.90% under C. Survivor Arguments CCA was 48.35% under both. Your concern about small denominators is illustrate by full-cohort Arguments: at the same 22.50% model accuracy, lift changes from 97.55 under H to approximately 8,700 under C. We therefore removed lift from Table 2 and report this sensitivity in the appendix.

4. Equivalence and identifiability

Appendix D.1 specifies geometric scoring within the declared structural mode. For Rules, we check whether the predicted identities can express the target affine map with legal arguments. For Arguments, we compare the map produced by the predicted parameters with the target map. Nonuniform scaling requires unequal axis factors and is disjoint with uniform scaling, so an equal-factor nonuniform prediction is invalid.

Geometric scoring does change survivor membership. Relative to symbolic scoring, the natural operand survivor count increased from 181 to 204 for Terra and from 171 to 191 for Kimi. The operand selection gap changed from 7.07 to 7.33 points for Terra and from 11.30 to 10.10 points for Kimi. We have updated the Figures and Tables affected with geometric results.

The identifiability concerns the construction and labeling of the synthetic dataset, rather than an assumption required by our proposed measurement framework. Nevertheless, we agree that ambiguity should be quantified: our audit identified five such cases among 4,100 examples (0.122%), which we disclose as a dataset limitation.

5. Stage-order sensitivity and ranking interpretation

As requested, we tested two alternative orders (Appendix H, Tables 17–18). The selection-gap point estimates retain their signs across the three tested orders, while magnitudes, coverage, and some interval-based conclusions vary. These results reinforce our framework’s requirement to declare the scoring order: changing the order can change the survivor population and hence the estimand, but does not invalidate its measurement.

We also would like to clarify that our framework's validity does not depend on a particular order, as our objective is to make population contrasts explicit and measurable. We do not aim at model diagnosis. Accordingly, the framework does not require selection gaps to have a particular magnitude or sign, or to exclude zero.

6. Repeated generations and interval interpretation

We agree that the reported intervals do not establish stability across repeated model generations. The reported intervals summarize item-level uncertainty in accuracies and population contrasts for the observed runs. We clarify that our objective is to study evaluation estimands and population selection, rather than diagnose model mechanisms or establish stable capability rankings. Fixed-prediction comparisons thus deliberately keep generated outputs constant and measure how changing the evaluated population changes the reported score.

Thank you again for the detailed feedback. We hope that our response addresses your concerns.

Dear Reviewer,

Thank you again for your detailed feedback. If you have an opportunity before the discussion closes, we would appreciate your assessment of the newly added pipeline audit, full operand oracle, and scoring and reference-policy clarifications.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.