- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper selects among sampled code solutions using their reasoning traces. It summarizes each trace, converts the summary into a graph, and trains a GNN to score candidates. It also uses the remaining selection errors to argue that reasoning alone has an information ceiling.
Critical strengths
I find the selection results interesting. The graph-based scorer improves over random selection and the text-embedding baseline, and I appreciate the effort to report variation across folds and seeds. The authors are also clear about how much ensembling helps and about the limited scope of the transfer experiment.
Critical weaknesses
I do not think the paper has established an information ceiling. A small score gap tells me that this scorer struggles to distinguish two candidates; it does not tell me that their original reasoning contains no distinguishing information. That distinction matters even more after two rounds of summarization and graph extraction. Likewise, the same judge doing better on code only shows what that judge can extract from each input. The paper's own GNN already extracts more from reasoning than the judge does. I therefore find the claim that no reader of the reasoning can close the remaining gap unjustified.
The graph result is promising, but I am less convinced by the explanation that half the gain comes from reasoning topology. The MLP comparison changes the representation and readout together, as the authors acknowledge. I cannot tell whether the benefit comes from the actual dependencies or simpler cues such as graph size and dead-end counts. The extraction prompt also explicitly pushes the model toward non-linear graphs, which makes me want some evidence that these edges reflect the trace rather than the prompt. A feature-matched non-graph scorer and a check of the extracted dependencies would make this much more convincing.
I also have trouble accepting the headline comparisons. SEP is evaluated on a different problem pool with its own anchors, so its lower normalized usefulness does not establish that this method closes twice as much of the gap. The cost argument is inconsistent too: the paper describes an order-of-magnitude token advantage and later says the judge reads the traces nine times over, while its own accounting gives 72 trace-reads for the judge versus roughly 64 plus short summaries for this pipeline. A smaller model may still be cheaper, but that needs to be demonstrated. These comparisons currently make the advantage sound stronger than the evidence supports.
Questions for authors
- What evidence separates information missing from the original reasoning from information lost or missed by the current pipeline? I would like the ceiling claim reconsidered on that basis.
- Does the GNN still help against a non-graph scorer given the same node features and comparable pooling? Have the extracted dependencies been checked against the original traces?
- How does the comparison with SEP change on the same problems and candidate pools, using common anchors?
- What are the measured selection costs, including both extraction passes? How do these reconcile with the claim that the judge reads the traces nine times over?
5 comments
Response to Reviewer qKQA (1/2)
Thank you for asking what our experiments establish about the information in a reasoning trace and the contribution of graph structure. We rechecked the main selection result using the saved predictions on held-out folds and ran the comparison you suggested with a control model that receives the same input features.
We evaluated the selector on 392 LiveCodeBench-v6 problems for which the retained Qwen3-8B candidates included both passing and failing programs. Each problem and all its candidates were held out from both training and checkpoint selection for the ensemble that scored them. The ensembles selected a passing program on 301 of 392 problems (76.79% pass@1), compared with an expected 67.30% under uniform random selection from the same candidate pools. Pooled normalized usefulness was U = 0.290, with a 95% confidence interval of 0.197 to 0.387 obtained by resampling whole problems while holding the trained ensembles fixed.
Q1. What evidence separates information missing from the original reasoning from information lost or missed by the current pipeline?
Short answer. Our experiments do not separate these explanations. We have therefore withdrawn the claim that we reached an information ceiling (abstract and Section 1) and revised Section 6 to present these observations as an analysis of where the evaluated selectors make errors.
When the selector assigns similar scores to two candidates, several explanations are consistent with that observation. The original traces may lack a cue that distinguishes the candidates, the cue may be lost during summarization or graph extraction, or the trained selector may fail to use it. A near-tie establishes only that this selector assigned similar scores. Likewise, the comparison between code and reasoning traces measures how effectively the particular judge uses each input.
Q2a. Does the GNN still help against a non-graph scorer given the same node features and comparable pooling?
Short answer. On the original split, the GNN retained an observed advantage over a control without message passing that receives the same inputs and uses the same pooling over nodes: the GNN selected a passing program on 301 of 392 problems (76.79%), compared with 287 (73.21%) for the control, a difference of 3.57 percentage points (two-sided exact sign test, p = 0.0595; Table 1 and Section 5, "Controlled comparison without message passing").
In Table 1, the full GNN achieved 76.79% pass@1 and the best MLP on pooled node embeddings achieved 72.63%. The difference of 4.16 percentage points is about 44% of the GNN's 9.49-point gain over random selection. This ratio was the basis for our wording "about half." Because this comparison changes both the candidate representation and the way the final score is computed, we have stated this in Section 5 ("Topology-free baseline") and removed that wording from Sections 1, 5, and 8.
To assess the contribution of learned operations on edges more directly, we trained the control model (no_mp), which processes each node independently with an MLP and uses the same sum and max pooling over nodes as the GNN. Both models receive the same node embeddings, flags, precomputed structural features, and metadata, including graph size, dead-end count, problem difficulty, and candidate index. We used the same loss, checkpoint selection rule, and ensemble size. The control disables message passing and edge pooling. We compared the models on the same held-out problems and candidate pools. The two architectures were trained with different random seeds.
The GNN alone succeeded on 31 problems, and the control alone succeeded on 17, which gives the observed advantage of 14 problems. This comparison measures the difference between the two trained architectures on matched candidate pools. Both retain structural information through their input features. In other words, both models read the extracted reasoning graph: the control through features computed from it, the GNN additionally through learned operations on its edges. Without message passing, the control still recovers 18.1% of the gap, more than the frontier judge reading the full traces (11.8%); message passing adds an observed 3.57 points on this split, a difference that our sample of 392 problems does not establish statistically (p = 0.0595).
We also tested the specific feature you suggested, the number of annotated dead ends. On the same 392 problems, selecting the candidate with the fewest dead ends yielded 270 passing programs, while selecting the one with the most yielded 231. The paired difference was 9.95 percentage points, with a 95% problem-bootstrap interval of 5.10 to 14.80 points. Relative to random selection, the fewest-dead-ends rule gained 1.58 points, with a 95% interval of -1.53 to +4.66 points. This analysis was exploratory (Section 5). We broke ties by selecting the first candidate in the original order.
Response to Reviewer qKQA (2/2)
Q2b. Have the extracted dependencies been checked against the original traces?
Short answer. Yes, through a manual qualitative inspection during development. For 24 reasoning traces, we traced the full transformation from the original chain of thought (CoT) through the step summary to the reasoning graph (GoT), for both intermediate and final versions of the extraction prompt, and for the edges we returned to the original text to assess whether it supported the proposed dependencies between reasoning steps.
The 24 traces come from eight problems across different domains, with three traces per problem. We compared nine models as extractors - Qwen3.5-9B, Qwen3.5-27B, Qwen3.5-35B, Qwen3.5-122B, Qwen3-8B, Qwen3-32B, gpt-oss-20b, gpt-oss-120b, and DeepSeek-V3.2.
For each graph, we checked the JSON conversion and graph connectivity. We matched each node to the cue that triggered its extraction and checked it against the original trace. The textual check of the edges matters because the extraction prompt explicitly encourages non-linear graphs. We selected Qwen3.5-27B on the basis of this qualitative comparison. We did not tally supported and unsupported edges, so we report the inspection procedure without assigning an edge-level accuracy rate. We have described the procedure in Appendix B ("Manual inspection of the extraction pipeline").
Q3. How does the comparison with SEP change on the same problems and candidate pools, using common anchors?
Short answer. We did not evaluate SEP on our saved candidate pools, so the relative performance of the two methods under common baselines remains unknown. We have withdrawn the claim that our method closes twice as much of the gap as SEP (Section 5, "Main result on LiveCodeBench-v6").
The published SEP result in Table 1 was obtained on a different set of 436 LeetCode problems, with its own candidate pools, random-selection baseline, and oracle score. We now list the published SEP result in a separate block of Table 1, apart from the comparisons conducted on our candidate pools.
Q4. What are the measured selection costs, including both extraction passes, and how do they reconcile with the claim that the judge reads the traces nine times over?
Short answer. For a problem with 64 candidates, our two extraction passes with Qwen3.5-27B process about 1.52M tokens (0.77M input and 0.76M generated), and the tournament judge processes about 0.96M tokens (0.74M input and 0.225M generated). Our pipeline therefore uses about 1.6 times as many tokens, almost entirely because of generation in the two passes, which includes the model's reasoning; at the same time, each of its calls needs about one eighth of the context of a judge call. We have withdrawn the claim of a tenfold token advantage (Section 5, "Main result on LiveCodeBench-v6"). The judge does not read each trace nine times: it makes 72 trace presentations across nine calls, and we have corrected the statement (Section 8).
We computed these figures from the API counters saved in an archived extraction run:
- First pass (original CoT -> step summary). A request averaged 10,546 input tokens, the CoT plus a 304-token summarization prompt, and 3,345 generated tokens, including the model's reasoning and the summary.
- Second pass (summary -> reasoning graph). A request averaged 1,456 input tokens, the summary plus the graph-construction prompt, and 8,457 generated tokens, including the model's reasoning and the JSON graph.
Over 64 candidates, this gives 64 * (10,546 + 3,345 + 1,456 + 8,457) ~1.52M tokens per problem. The largest single item is generation during graph construction, about 0.54M tokens per problem.
The tournament judge processes eight groups of eight traces and then compares the eight winners. Each of the nine calls contains eight traces of about 10,242 tokens on average and a 209-token tournament prompt, about 82,000 input tokens in total, and the judge generated about 25,000 tokens per call with reasoning enabled. This gives about 0.74M input and 0.225M generated tokens per problem. We estimated the judge's input from trace lengths in the Qwen tokenizer, so the judge's own tokenizer may give somewhat different counts. The per-problem figures for our pipeline assume one request per candidate in each pass, without retries.
The two methods also differ in how much context a single call requires. A judge call holds about 82,000 input tokens on average, roughly eight times the 10,546-token average of our first pass. Our extractor reads one trace per call, so the 64 calls of each pass are independent and can run in parallel with separate contexts. We have added these measurements to the "Selection compute" paragraphs of Section 5 and Appendix C.
I have received the committee’s message about possible AI use in reviews and would like to clarify how I used it for this paper.
I used AI to prepare a bilingual copy of the paper, and discuss its method and results. I also check whether my concerns were supported by the manuscript. It also drafted English wording and questions for me to consider.
I take responsibility for what I have writen and am open to the authors’ response. I will follow the discussion and update my assessment if their response addresses my concerns.