acceptodds
Under review as a conference paper at ICLR 2027

How Far Can Reasoning Alone Take You? Graph-Based Candidate Selection for Test-Time Scaling on Code

Abstract

Large reasoning models spend most of their test-time compute on long chains of thought, and on competitive programming the quality of that reasoning — rather than implementation ability — is the dominant bottleneck. Given sampled candidates per problem, the question is which one to trust. We ask how much of that selection problem is solvable from the reasoning trace alone: no code execution, no answer consensus, no token probabilities. We condense each trace into a summary, lift the summary into a reasoning graph whose topology encodes steps, dependencies and case analysis, and train a graph neural network (GNN) to predict correctness from the graph. On LiveCodeBench-v6, under a leakage-free 10-fold cross-test protocol in which every problem is scored only by models that never saw it during training or checkpoint selection, per-fold ensembles recover 0.29 of the gap (+9.5 points absolute over random selection), against 0.12 for a frontier judge reading the same traces and 0.16 for a topology-free MLP over the same summary embeddings; single models recover 0.19, so seed-ensembling — effective here because a small training set and aggressive dropout yield weakly correlated models — accounts for a third of the effect, and the worst of the ten folds still recovers 0.16. A comparison of seven checkpoint-selection rules shows that selecting on a smoothed validation curve dominates raw peak selection in both mean and worst-case usefulness. As a transfer example, on a harder LiveCodeBench-Pro subset the classifier–selector cascade recovers . We then characterize the ceiling of reasoning-only selection: 94.5% of incorrect candidates textually claim success, and most residual errors are near-ties where candidates share an algorithm and differ only in implementation details absent from the trace — so much so that the same judge, held fixed, recovers three times as much of the gap from the candidates' code as from their reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large reasoning models, test-time compute, best-of-N selection, chain-of-thought, reasoning graphs, graph neural networks, code generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qKQAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper selects among sampled code solutions using their reasoning traces. It summarizes each trace, converts the summary into a graph, and trains a GNN to score candidates. It also uses the remaining selection errors to argue that reasoning alone has an information ceiling.

Critical strengths

I find the selection results interesting. The graph-based scorer improves over random selection and the text-embedding baseline, and I appreciate the effort to report variation across folds and seeds. The authors are also clear about how much ensembling helps and about the limited scope of the transfer experiment.

Critical weaknesses

I do not think the paper has established an information ceiling. A small score gap tells me that this scorer struggles to distinguish two candidates; it does not tell me that their original reasoning contains no distinguishing information. That distinction matters even more after two rounds of summarization and graph extraction. Likewise, the same judge doing better on code only shows what that judge can extract from each input. The paper's own GNN already extracts more from reasoning than the judge does. I therefore find the claim that no reader of the reasoning can close the remaining gap unjustified.

The graph result is promising, but I am less convinced by the explanation that half the gain comes from reasoning topology. The MLP comparison changes the representation and readout together, as the authors acknowledge. I cannot tell whether the benefit comes from the actual dependencies or simpler cues such as graph size and dead-end counts. The extraction prompt also explicitly pushes the model toward non-linear graphs, which makes me want some evidence that these edges reflect the trace rather than the prompt. A feature-matched non-graph scorer and a check of the extracted dependencies would make this much more convincing.

I also have trouble accepting the headline comparisons. SEP is evaluated on a different problem pool with its own anchors, so its lower normalized usefulness does not establish that this method closes twice as much of the gap. The cost argument is inconsistent too: the paper describes an order-of-magnitude token advantage and later says the judge reads the traces nine times over, while its own accounting gives 72 trace-reads for the judge versus roughly 64 plus short summaries for this pipeline. A smaller model may still be cheaper, but that needs to be demonstrated. These comparisons currently make the advantage sound stronger than the evidence supports.

Questions for authors
  1. What evidence separates information missing from the original reasoning from information lost or missed by the current pipeline? I would like the ceiling claim reconsidered on that basis.
  2. Does the GNN still help against a non-graph scorer given the same node features and comparable pooling? Have the extracted dependencies been checked against the original traces?
  3. How does the comparison with SEP change on the same problems and candidate pools, using common anchors?
  4. What are the measured selection costs, including both extraction passes? How do these reconcile with the claim that the judge reads the traces nine times over?
5 comments

I have received the committee’s message about possible AI use in reviews and would like to clarify how I used it for this paper.

I used AI to prepare a bilingual copy of the paper, and discuss its method and results. I also check whether my concerns were supported by the manuscript. It also drafted English wording and questions for me to consider.

I take responsibility for what I have writen and am open to the authors’ response. I will follow the discussion and update my assessment if their response addresses my concerns.

AuthorsOpenReview ↗

Response to Reviewer qKQA (1/2)

Thank you for asking what our experiments establish about the information in a reasoning trace and the contribution of graph structure. We rechecked the main selection result using the saved predictions on held-out folds and ran the comparison you suggested with a control model that receives the same input features.

We evaluated the selector on 392 LiveCodeBench-v6 problems for which the retained Qwen3-8B candidates included both passing and failing programs. Each problem and all its candidates were held out from both training and checkpoint selection for the ensemble that scored them. The ensembles selected a passing program on 301 of 392 problems (76.79% pass@1), compared with an expected 67.30% under uniform random selection from the same candidate pools. Pooled normalized usefulness was U = 0.290, with a 95% confidence interval of 0.197 to 0.387 obtained by resampling whole problems while holding the trained ensembles fixed.

Q1. What evidence separates information missing from the original reasoning from information lost or missed by the current pipeline?

Short answer. Our experiments do not separate these explanations. We have therefore withdrawn the claim that we reached an information ceiling (abstract and Section 1) and revised Section 6 to present these observations as an analysis of where the evaluated selectors make errors.

When the selector assigns similar scores to two candidates, several explanations are consistent with that observation. The original traces may lack a cue that distinguishes the candidates, the cue may be lost during summarization or graph extraction, or the trained selector may fail to use it. A near-tie establishes only that this selector assigned similar scores. Likewise, the comparison between code and reasoning traces measures how effectively the particular judge uses each input.

Q2a. Does the GNN still help against a non-graph scorer given the same node features and comparable pooling?

Short answer. On the original split, the GNN retained an observed advantage over a control without message passing that receives the same inputs and uses the same pooling over nodes: the GNN selected a passing program on 301 of 392 problems (76.79%), compared with 287 (73.21%) for the control, a difference of 3.57 percentage points (two-sided exact sign test, p = 0.0595; Table 1 and Section 5, "Controlled comparison without message passing").

In Table 1, the full GNN achieved 76.79% pass@1 and the best MLP on pooled node embeddings achieved 72.63%. The difference of 4.16 percentage points is about 44% of the GNN's 9.49-point gain over random selection. This ratio was the basis for our wording "about half." Because this comparison changes both the candidate representation and the way the final score is computed, we have stated this in Section 5 ("Topology-free baseline") and removed that wording from Sections 1, 5, and 8.

To assess the contribution of learned operations on edges more directly, we trained the control model (no_mp), which processes each node independently with an MLP and uses the same sum and max pooling over nodes as the GNN. Both models receive the same node embeddings, flags, precomputed structural features, and metadata, including graph size, dead-end count, problem difficulty, and candidate index. We used the same loss, checkpoint selection rule, and ensemble size. The control disables message passing and edge pooling. We compared the models on the same held-out problems and candidate pools. The two architectures were trained with different random seeds.

The GNN alone succeeded on 31 problems, and the control alone succeeded on 17, which gives the observed advantage of 14 problems. This comparison measures the difference between the two trained architectures on matched candidate pools. Both retain structural information through their input features. In other words, both models read the extracted reasoning graph: the control through features computed from it, the GNN additionally through learned operations on its edges. Without message passing, the control still recovers 18.1% of the gap, more than the frontier judge reading the full traces (11.8%); message passing adds an observed 3.57 points on this split, a difference that our sample of 392 problems does not establish statistically (p = 0.0595).

We also tested the specific feature you suggested, the number of annotated dead ends. On the same 392 problems, selecting the candidate with the fewest dead ends yielded 270 passing programs, while selecting the one with the most yielded 231. The paired difference was 9.95 percentage points, with a 95% problem-bootstrap interval of 5.10 to 14.80 points. Relative to random selection, the fewest-dead-ends rule gained 1.58 points, with a 95% interval of -1.53 to +4.66 points. This analysis was exploratory (Section 5). We broke ties by selecting the first candidate in the original order.

AuthorsOpenReview ↗

Response to Reviewer qKQA (2/2)

Q2b. Have the extracted dependencies been checked against the original traces?

Short answer. Yes, through a manual qualitative inspection during development. For 24 reasoning traces, we traced the full transformation from the original chain of thought (CoT) through the step summary to the reasoning graph (GoT), for both intermediate and final versions of the extraction prompt, and for the edges we returned to the original text to assess whether it supported the proposed dependencies between reasoning steps.

The 24 traces come from eight problems across different domains, with three traces per problem. We compared nine models as extractors - Qwen3.5-9B, Qwen3.5-27B, Qwen3.5-35B, Qwen3.5-122B, Qwen3-8B, Qwen3-32B, gpt-oss-20b, gpt-oss-120b, and DeepSeek-V3.2.

For each graph, we checked the JSON conversion and graph connectivity. We matched each node to the cue that triggered its extraction and checked it against the original trace. The textual check of the edges matters because the extraction prompt explicitly encourages non-linear graphs. We selected Qwen3.5-27B on the basis of this qualitative comparison. We did not tally supported and unsupported edges, so we report the inspection procedure without assigning an edge-level accuracy rate. We have described the procedure in Appendix B ("Manual inspection of the extraction pipeline").

Q3. How does the comparison with SEP change on the same problems and candidate pools, using common anchors?

Short answer. We did not evaluate SEP on our saved candidate pools, so the relative performance of the two methods under common baselines remains unknown. We have withdrawn the claim that our method closes twice as much of the gap as SEP (Section 5, "Main result on LiveCodeBench-v6").

The published SEP result in Table 1 was obtained on a different set of 436 LeetCode problems, with its own candidate pools, random-selection baseline, and oracle score. We now list the published SEP result in a separate block of Table 1, apart from the comparisons conducted on our candidate pools.

Q4. What are the measured selection costs, including both extraction passes, and how do they reconcile with the claim that the judge reads the traces nine times over?

Short answer. For a problem with 64 candidates, our two extraction passes with Qwen3.5-27B process about 1.52M tokens (0.77M input and 0.76M generated), and the tournament judge processes about 0.96M tokens (0.74M input and 0.225M generated). Our pipeline therefore uses about 1.6 times as many tokens, almost entirely because of generation in the two passes, which includes the model's reasoning; at the same time, each of its calls needs about one eighth of the context of a judge call. We have withdrawn the claim of a tenfold token advantage (Section 5, "Main result on LiveCodeBench-v6"). The judge does not read each trace nine times: it makes 72 trace presentations across nine calls, and we have corrected the statement (Section 8).

We computed these figures from the API counters saved in an archived extraction run:

  • First pass (original CoT -> step summary). A request averaged 10,546 input tokens, the CoT plus a 304-token summarization prompt, and 3,345 generated tokens, including the model's reasoning and the summary.
  • Second pass (summary -> reasoning graph). A request averaged 1,456 input tokens, the summary plus the graph-construction prompt, and 8,457 generated tokens, including the model's reasoning and the JSON graph.

Over 64 candidates, this gives 64 * (10,546 + 3,345 + 1,456 + 8,457) ~1.52M tokens per problem. The largest single item is generation during graph construction, about 0.54M tokens per problem.

The tournament judge processes eight groups of eight traces and then compares the eight winners. Each of the nine calls contains eight traces of about 10,242 tokens on average and a 209-token tournament prompt, about 82,000 input tokens in total, and the judge generated about 25,000 tokens per call with reasoning enabled. This gives about 0.74M input and 0.225M generated tokens per problem. We estimated the judge's input from trace lengths in the Qwen tokenizer, so the judge's own tokenizer may give somewhat different counts. The per-problem figures for our pipeline assume one request per candidate in each pass, without retries.

The two methods also differ in how much context a single call requires. A judge call holds about 82,000 input tokens on average, roughly eight times the 10,546-token average of our first pass. Our extractor reads one trace per call, so the 64 calls of each pass are independent and can run in parallel with separate contexts. We have added these measurements to the "Selection compute" paragraphs of Section 5 and Appendix C.

Reviewer_34A6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a GNN-based approach to selecting correct code candidates using only their reasoning traces, without code execution. Experiments on LiveCodeBench demonstrate improvements over several baselines. The paper also investigates model ensembling, checkpoint selection, and the limitations of reasoning-only selection.

Critical strengths
  1. Interesting research question. The paper studies how much signal reasoning traces alone provide for selecting correct code candidates, without inspecting or executing the code.

  2. Useful evaluation and analysis. The problem-grouped cross-validation and analyses of checkpoint selection and seed ensembling provide useful insights into the stability of the learned selector.

Critical weaknesses
  1. Limited evidence of cross-model generalization. All reasoning traces are generated by Qwen3-8B, and the transfer experiment keeps the generator fixed. It remains unclear whether the GNN learns broadly applicable signals of reasoning quality or generator-specific patterns. How well does the trained selector transfer to traces from another model without retraining?

  2. The broader conclusions exceed the empirical scope. The evaluation is restricted to competitive programming with a single generator. While this is a reasonable setting for the proposed method, it provides limited support for the paper’s broader claims about the information ceiling of reasoning-only selection.

  3. Moderate empirical gains. The final ensemble improves over the strongest topology-free MLP by 4.16 percentage points on the filtered LiveCodeBench subset. This is a useful but moderate gain. Stronger evidence of robustness and better-controlled attribution to graph structure would make the empirical contribution more convincing.

Questions for authors

See the weakness section

1 comment
AuthorsOpenReview ↗

Response to Reviewer 34A6

Thank you for asking how well the selector transfers to other models, how robust the observed gain is, and how much of it can be attributed to graph structure.

W1. How well does the trained selector transfer to traces from another model without retraining?

Short answer. This has yet to be tested. We evaluated the selector only on reasoning traces generated by Qwen3-8B, and we have explicitly stated this limitation in Section 7. We see this as a question for future work: the answer will show whether the reasoning-graph structure that the selector relies on is a property of long chains of thought in general.

W2. Scope of the conclusions about an information ceiling

Short answer. We have withdrawn the claim that we reached an information ceiling (abstract, Section 1, and Section 6). The original trace may contain cues about correctness that are lost during summarization or graph construction, or that the trained model does not use.

The main scientific contribution of our work is to quantify how effectively graph representations of chains of thought (CoTs) can be used to predict the correctness of the resulting program and to select among alternative solutions. In other words, we aim to extract a signal about a program's correctness from the reasoning that produced it rather than from its code. Our method for extracting and processing the signal in Qwen3-8B reasoning traces increased pass@1 from 67.30% under random selection to 76.79%, a gain of 9.49 percentage points. The selector made an incorrect choice on 23.21% of problems, compared with an expected 32.70% under random selection. This corresponds to a 29% relative reduction in selection errors. We consider this a substantial reduction for a selector that uses reasoning graphs to choose among solutions to previously unseen problems.

W3. Magnitude and robustness of the gain, and controlled attribution to graph structure

Short answer. The gain over random selection held on every held-out fold: on each of the ten folds, the ensemble exceeded the random-selection baseline calculated for that fold, and normalized usefulness was U = 0.290, with a 95% confidence interval of 0.197 to 0.387. In a controlled comparison, a model that received the same node embeddings and precomputed graph features, but with message passing and edge pooling disabled, achieved 73.21% pass@1 on the original split, 3.57 percentage points below the GNN. We have added this comparison to Table 1 and Section 5 ("Controlled comparison without message passing").

On the original set of 392 problems, the GNN ensemble achieved 76.79% pass@1. The best MLP achieved 72.63%, and the random-selection baseline was 67.30%. We obtained the interval for U by bootstrapping over problems while holding the trained models fixed.

On the same candidate pools, we examined whether individual graph properties could be used to select solutions. In this exploratory analysis, selecting the candidate with the fewest nodes annotated as dead ends yielded 68.88% pass@1, while selecting the one with the most yielded 58.93%. The paired difference was 9.95 percentage points, with a 95% problem-bootstrap interval of 5.10 to 14.80 points. Selecting the candidate with the most dead ends performed 8.37 points below random selection. Relative to random selection, the fewest-dead-ends rule gained 1.58 points and the maximum-graph-density rule gained 2.60 points. Both confidence intervals for the gain included zero. When several candidates tied for the best value of a feature, we selected the first in the original order. We have added this analysis to Section 5 ("Single graph statistics as selectors").

These results show that the number of annotated dead ends is associated with whether the selected program passes the tests. To estimate the additional contribution of learned operations on edges, we compared the GNN with the control model described above.

Reviewer_d842OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies candidate code selection in competitive programming and formulates a task in which candidate correctness is assessed solely from the reasoning process. It proposes a pipeline that summarizes chain-of-thought, extracts reasoning graphs, and scores candidates with a graph neural network. Specifically, Qwen3.5-27B condenses each chain of thought into step summaries, which are represented as directed graphs with typed nodes and edges; a GINE network then assigns a score to each candidate.

Critical strengths

1.The task of assessing candidate correctness solely from the reasoning process is meaningful: it can reveal whether a model selects candidates based on sound reasoning. 2.The authors’ input constraint is a novel and valuable aspect of the study, enabling them to examine whether reasoning text generated through chain-of-thought contains reliable signals of correctness. 3.The authors split the data by problem rather than by candidate, ensuring that each test problem is scored by models that have not encountered it during training. This evaluation protocol provides some evidence of the models’ ability to generalize across problem types.

Critical weaknesses

1.The paper’s primary contribution appears to lie in its problem formulation and input constraint, while its technical approach largely builds on existing methods. However, the paper does not clearly explain how competitive programming differs from other code-generation settings or whether the proposed method generalizes to other code-generation tasks. The authors should also clarify whether the technical approach itself offers any methodological novelty. 2.The baselines are primarily general selection paradigms rather than advanced methods designed for competitive programming. As a result, the comparisons do not establish the proposed method’s effectiveness relative to the relevant state of the art.

Questions for authors

1.How does competitive programming differ from other code-generation settings, and why does it warrant a separate study? The paper does not clearly identify the challenges unique to this setting. The authors should also discuss whether their method generalizes to general-purpose code generation. 2.The evaluation lacks comparisons with advanced methods tailored to competitive programming. The authors should provide stronger evidence of the proposed method’s effectiveness and clarify its advantages over relevant approaches.

1 comment
AuthorsOpenReview ↗

Response to Reviewer d842

Thank you for recognizing the value of studying selection from reasoning traces and of evaluating the selector on held-out problems.

Q1. How does competitive programming differ from other code-generation settings, why study it separately, and does the method generalize?

Short answer. We chose competitive programming because it gives us a direct way to measure whether a reasoning trace helps identify a correct program: the properties listed below occur together in this setting, and correctness can be evaluated independently of the selector. Unlike software-engineering tasks such as SWE-bench, checking the traces and the extracted graphs by hand does not require expertise in several application areas. Generalization to general-purpose code generation remains untested, and we have stated this limit in Section 7.

In competitive programming, candidates for the same problem are evaluated on the same hidden tests under time and memory limits. On these problems, the generator produces long chains of thought (CoTs), which often record the choice of algorithm, its expected complexity, and the treatment of edge cases. An implementation error can still cause a program to fail even when its reasoning describes a sound algorithm. Different correct programs can also implement the same algorithm in different ways, which makes voting on the program text unreliable.

We also chose competitive programming so that, when improving the pipeline, we could manually match each original CoT to its problem, and then each reasoning graph (GoT) to its CoT, without much effort. This manual inspection is described in our response to Reviewer qKQA (Q2b).

Our experiments use problems unseen during training and reasoning traces from Qwen3-8B. Transfer to other generators and to repository-level tasks also remains untested, and we have stated these limits in Section 7 as well.

W1. Does the technical approach itself offer methodological novelty?

Short answer. Our main scientific contribution lies in the measurement rather than in the graph layers, which are established (GINE): we quantify how effectively CoT-derived representations can predict the correctness of the resulting program and support selection among alternative solutions on previously unseen problems.

We summarize each trace, represent the dependencies between reasoning steps as a typed graph, and score the candidates with GINE layers.

On 392 LiveCodeBench-v6 problems containing both passing and failing candidates, the method increased pass@1 from 67.30% under uniform random selection to 76.79%, a gain of 9.49 percentage points. This closes 29% of the gap between random selection and an oracle that always picks a passing candidate. We now report this as the performance of the method evaluated in the paper and have withdrawn the claim that it reaches an information ceiling (abstract, Section 1, and Section 6).

Q2 / W2. Comparisons with advanced methods tailored to competitive programming

Short answer. Our setting differs from these methods in where the signal about correctness comes from: we extract it from a solution's chain of thought rather than from its code, whereas S*, SEP, and CodeT use additional evidence obtained from the code itself, by executing candidates or evaluating them on tests. We have not compared these methods with ours on the same problems and candidate pools; the directly measured result is the gain in selection accuracy under this input restriction. Our focus is not on maximizing selection accuracy but on studying how much signal about a solution's correctness can be extracted from its long reasoning trace.

S* uses generated inputs to distinguish candidates through execution, SEP uses public examples and symbolic execution, and CodeT evaluates candidates on generated tests. Our selector scores the candidates using representations derived from their reasoning. Hidden tests establish the outcome against which a selection is evaluated.

The reported SEP result comes from a different problem set with its own random and oracle baselines. We now list it separately in Table 1 and have removed the claim that our method closes twice as much of the gap as SEP (Section 5, "Main result on LiveCodeBench-v6").

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.