- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SliceGraph is a post-hoc framework for analyzing multiple chain-of-thought trajectories for one problem and model. It represents text slices by sparse MLP activation keys, builds a mutual-nearest-neighbor graph, and extracts biconnected blocks. Runs are clustered into structural route families according to their block-visitation footprints; correct runs with the same answer but different families are termed structural process isomers. The paper reports route multiplicity across 954 primary problem-model cells, configuration and held-out analyses, human semantic audits, and additional model replications. A label-free route-collapse flag identifies less reliable consensus, while family-based voting does not improve self-consistency.
Critical strengths
-
The paper addresses a useful gap between answer agreement and process diversity. The graph, blocks, and families provide an inspectable description of multiple reasoning traces. The separation between correctness-conditioned diversity analysis and label-free consensus auditing is particularly helpful, and the secondary reward overlay does not determine the core family partition.
-
The validation is extensive and the conclusions are carefully qualified. The paper includes human annotation, configuration sweeps, correct-run-count controls, genuine held-out atlases, and additional architecture/scale replications. It clearly distinguishes structural families from human-named strategies and reports both unstable family identities and the negative result on family-based voting. These disclosures make the evidence substantially easier to assess.
Critical weaknesses
-
Family boundaries lack independent semantic calibration. Isomer rates depend on family sizes and vary from 36.5–91.5% across resolution settings. Yet the semantic audit covers only eight selected family-rich cases, with 10/31 across-family pairs judged different-strategy. This leaves unclear which scale captures meaningful process distinctions and whether other boundaries reflect local operations, wording, or partition effects. The paper acknowledges this ambiguity, but broader sampling and full-trace validation are needed to resolve it.
-
Process fidelity remains confounded by information loss and graph construction. Footprints omit visitation order and repetition, while biconnectivity does not ensure semantic coherence throughout a block; Appendix E.2 mainly checks nearby within-block pairs. Although bridge removal reduces length-related fragmentation, the residual length-variance ratio of 0.460 lacks a clear definition and reference comparison. Subsampling to the 2,600-slice cap before graph construction can also change neighborhoods and blocks. Ordered-sequence comparisons, length/position-matched controls, and cap sensitivity are needed to establish what process differences survive these choices.
-
The activation representation lacks independent validation. Rescaling an up-projection unit and inversely rescaling its down-projection column can preserve the MLP function while changing the globally ranked activation keys; the effect on families is untested. MiniLM’s collapse within the fixed scaffold also does not establish inferior process measurement. Different representations should be calibrated and compared against the same semantic targets, rather than assessed primarily through family multiplicity.
-
The risk flag’s incremental auditing value is not established. Lower accuracy in route-collapsed cohorts and vote-share stratification are useful evidence. However, the binary flag lacks evaluation on held-out problems against comparably calibrated vote share, vote margin, answer entropy, and length baselines. The negative AUROC result for family-support scoring does not answer this separate question. Evaluation should establish the flag’s additional information within the stated retrospective setting.
Questions for authors
-
Semantic calibration: Can graph and clustering settings be fixed using independent annotated or verifiable examples? Would a stratified random audit using full traces support the same conclusions when strategy, local-operation, and wording differences are labeled separately?
-
Process fidelity and confounds: Do the findings persist with ordered footprints, length/position matching, and different slice caps? Please define the length-variance ratio and its reference comparison, and check semantic coherence between distant slices within a block. For dynamics, can a null preserving footprints, length, and visitation counts isolate ordering effects?
-
Representation validity: How sensitive are families to function-preserving unit rescaling? How do normalized or output-contribution-based activations and separately calibrated text representations compare on the same external semantic judgments?
-
Incremental risk information: On held-out problems at matched coverage, does the collapse flag improve risk estimation over vote and length baselines? Please report cohort and stratum sizes and problem-clustered uncertainty within the stated retrospective 64-run-atlas setting.