acceptodds
Under review as a conference paper at ICLR 2027

SliceGraph: Mapping Process Isomers in Multi-Run Chain-of-Thought Reasoning

Abstract

Multi-run chain-of-thought reasoning is usually collapsed to final-answer aggregates, which discard how sampled trajectories share, split, and rejoin through intermediate computation. We propose **SliceGraph**, a post-hoc problem-model-cell graph built by mutual-*k*NN over sparse activation-key Jaccard similarity between CoT slices, and treat it as a *measurement object* for process geometry rather than as a decoding program. Across sampled CoT ensembles from three primary 4B/8B models on math and science benchmarks, blinded annotation supports SliceGraph biconnected components as shared reasoning-state units and process families as within-family strategy-coherent route units. In 85.5% of 954 problem-model cells, correct CoTs sharing the same normalized answer split into multiple process families; among cells with at least two such runs, 76.6% of run pairs are cross-family on average. We call such same-answer, family-divergent correct trajectories *process isomers*. A label-seeded reward field provides a separate value-landscape layer: success-associated regions often split into disconnected high-value cores, and route families specialize over these core footprints rather than merely duplicating one another. A typed-state transition analysis further shows that process families navigate the same atlas with distinct transition kernels under matched null controls. Representation ablations, a cross-architecture replication, and two cross-scale replications support the robustness of the route-family scaffold, showing that final-answer aggregation overlooks this structured multi-route process geometry.

Primary area:
interpretability and explainable AI
Keywords:
Chain-of-Thought Reasoning, Multi-Run Reasoning, Reasoning Trajectories, Process Geometry, SliceGraph, Process Families, Representation Analysis, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jVREOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SliceGraph is a post-hoc framework for analyzing multiple chain-of-thought trajectories for one problem and model. It represents text slices by sparse MLP activation keys, builds a mutual-nearest-neighbor graph, and extracts biconnected blocks. Runs are clustered into structural route families according to their block-visitation footprints; correct runs with the same answer but different families are termed structural process isomers. The paper reports route multiplicity across 954 primary problem-model cells, configuration and held-out analyses, human semantic audits, and additional model replications. A label-free route-collapse flag identifies less reliable consensus, while family-based voting does not improve self-consistency.

Critical strengths
  1. The paper addresses a useful gap between answer agreement and process diversity. The graph, blocks, and families provide an inspectable description of multiple reasoning traces. The separation between correctness-conditioned diversity analysis and label-free consensus auditing is particularly helpful, and the secondary reward overlay does not determine the core family partition.

  2. The validation is extensive and the conclusions are carefully qualified. The paper includes human annotation, configuration sweeps, correct-run-count controls, genuine held-out atlases, and additional architecture/scale replications. It clearly distinguishes structural families from human-named strategies and reports both unstable family identities and the negative result on family-based voting. These disclosures make the evidence substantially easier to assess.

Critical weaknesses
  1. Family boundaries lack independent semantic calibration. Isomer rates depend on family sizes and vary from 36.5–91.5% across resolution settings. Yet the semantic audit covers only eight selected family-rich cases, with 10/31 across-family pairs judged different-strategy. This leaves unclear which scale captures meaningful process distinctions and whether other boundaries reflect local operations, wording, or partition effects. The paper acknowledges this ambiguity, but broader sampling and full-trace validation are needed to resolve it.

  2. Process fidelity remains confounded by information loss and graph construction. Footprints omit visitation order and repetition, while biconnectivity does not ensure semantic coherence throughout a block; Appendix E.2 mainly checks nearby within-block pairs. Although bridge removal reduces length-related fragmentation, the residual length-variance ratio of 0.460 lacks a clear definition and reference comparison. Subsampling to the 2,600-slice cap before graph construction can also change neighborhoods and blocks. Ordered-sequence comparisons, length/position-matched controls, and cap sensitivity are needed to establish what process differences survive these choices.

  3. The activation representation lacks independent validation. Rescaling an up-projection unit and inversely rescaling its down-projection column can preserve the MLP function while changing the globally ranked activation keys; the effect on families is untested. MiniLM’s collapse within the fixed scaffold also does not establish inferior process measurement. Different representations should be calibrated and compared against the same semantic targets, rather than assessed primarily through family multiplicity.

  4. The risk flag’s incremental auditing value is not established. Lower accuracy in route-collapsed cohorts and vote-share stratification are useful evidence. However, the binary flag lacks evaluation on held-out problems against comparably calibrated vote share, vote margin, answer entropy, and length baselines. The negative AUROC result for family-support scoring does not answer this separate question. Evaluation should establish the flag’s additional information within the stated retrospective setting.

Questions for authors
  1. Semantic calibration: Can graph and clustering settings be fixed using independent annotated or verifiable examples? Would a stratified random audit using full traces support the same conclusions when strategy, local-operation, and wording differences are labeled separately?

  2. Process fidelity and confounds: Do the findings persist with ordered footprints, length/position matching, and different slice caps? Please define the length-variance ratio and its reference comparison, and check semantic coherence between distant slices within a block. For dynamics, can a null preserving footprints, length, and visitation counts isolate ordering effects?

  3. Representation validity: How sensitive are families to function-preserving unit rescaling? How do normalized or output-contribution-based activations and separately calibrated text representations compare on the same external semantic judgments?

  4. Incremental risk information: On held-out problems at matched coverage, does the collapse flag improve risk estimation over vote and length baselines? Please report cohort and stratum sizes and problem-clustered uncertainty within the stated retrospective 64-run-atlas setting.

Reviewer_gYCKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

SliceGraph is a post-hoc analysis pipeline that groups multiple CoT runs into structural route families using sparse activation similarity and graph decomposition. It shows that correct runs with the same answer often follow structurally different routes.

Critical strengths
  1. It provides an interesting and carefully designed pipeline for studying process diversity beyond final-answer agreement.
  2. It includes extensive robustness checks, human annotation, and replications across models and benchmarks.
Critical weaknesses
  1. The exact route families are sensitive to graph and clustering configurations and only partially align with human-interpretable reasoning strategies.
  2. The practical utility of the proposed signal remains limited: incorporating route families into majority voting does not improve performance, making its practical value unclear.
Questions for authors

How should the SliceGraph configuration be selected in practice, given that route-family membership changes substantially with graph scale and clustering resolution?

Reviewer_4swLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SliceGraph, a post-hoc framework for analyzing structural diversity in CoT reasoning trajectories which share the same final answer. Experiments were based on 3 models and 5 benchmarks, which show that multiple distinct route families frequently exist among correct runs producing the same answer. The paper also investigates the robustness and interpretability of these families.

Critical strengths
  1. the paper is well structured overall, provides a comprehensive description of the method, and is easy to follow

  2. I like the fact that the paper reports negative results quite transparenlty, including the lack of gains from family-aware voting and the limited alignment between cross-family boundaries and human strategy judgments.

Critical weaknesses
  1. The evidence does not seem to sufficiently support the paper’s central claim about reasoning diversity. Interpreting cross-family separation as different reasoning routes requires semantic validation. However, §4.2 mainly establishes within-family coherence: only 10/21 decisive cross-family pairs were judged to use different strategies. High within-family coherence cannot rule out over-fragmentation, where a single strategy is split into multiple families. Thus, I am not fully convinced about the prevalence of semantically distinct reasoning routes.

  2. I would argue that identifying a subgroup with lower accuracy does not establish a substantial contribution, even if it outperforms simpler baselines. Also, the paper does not show how this observation improves downstream decisions.

Questions for authors
  1. How is “the same reasoning operation” defined?
  2. What substantive advantages does this framework offer over simple text-based analyses or approaches such as GraphReason?
  3. Why were only 47.6% of decisive cross-family pairs judged to use different strategies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.