acceptodds
Under review as a conference paper at ICLR 2027

Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models

Abstract

The rapid capability gains of frontier language models are widely attributed to improved reasoning abilities, yet this cannot be verified as raw chain-of-thought (CoT) traces in closed-source systems are hidden. By registering a simple custom tool through a standard API feature, we induce frontier models to externalize intermediate reasoning. Because these traces may reflect post-hoc rationalization rather than genuine reasoning, we first evaluate against native CoT on open-source models and extend to closed-source frontier models including GPT-6 Astra. We find that the extracted reasoning matches native reasoning performance and substantially outperforms no-reasoning baselines, across competition mathematics, science, and code generation. We then provide an account of the structural properties of frontier models' intermediate reasoning behavior. Across token efficiency, reasoning-step types, and induced reasoning graphs, we identify systematic differences in how models externalize, compress, and organize reasoning. We find that Astra exhibits token-efficient directed reasoning, selecting a correct trajectory earlier, while resolving elementary steps internally and externalizing only crucial reasoning. These findings provide a behavioral lens on frontier-model reasoning beyond benchmark scores.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Chain-of-Thought Reasoning, Reasoning Trace Extraction, Reasoning Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vFgbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Forced-Reasoning, a tool-calling protocol that elicits visible intermediate reasoning through a custom tool. It compares induced and native reasoning on open models and evaluates closed models on mathematics, multidisciplinary questions, and code generation. It then characterizes the induced traces using token counts, zlib compressibility, reasoning-activity annotations, reconstructed trees, and cross-model reuse. The main finding is that Astra produces shorter, less repetitive, and more compactly organized visible traces, whose usefulness varies across recipient models. The paper interprets these observations in terms of selective externalization and reasoning efficiency.

Critical strengths
  1. The protocol provides a useful and inspectable way to study elicited reasoning behavior. Validation includes ten-round experiments on two open models, native-to-native and native-to-forced lexical comparisons, and functional annotations. The same-instruction, no-tool controls for Sol also help distinguish the tool protocol from simply requesting a longer answer.

  2. The empirical characterization is broad and acknowledges important limitations. The paper combines performance, local expression, activity composition, tree structure, and trace reuse across several task types. It distinguishes tool-stage from whole-interaction native-token usage and explicitly acknowledges that behavioral similarity does not establish identity with internal computation, and that in-context reuse is not distillation.

Critical weaknesses
  1. Behavioral validation does not establish recovery of native reasoning. Forced tool selection and replay change the reasoning conditions. Accuracy, lexical overlap, and coarse activity similarity support a useful proxy, but not the abstract’s claims that Astra selects the correct trajectory earlier or resolves elementary steps internally. Appendix K also finds differences in verification, monitoring, and exploration. The limitation in Section 5 should more directly constrain these central interpretations.

  2. Model-specific protocols confound the comparisons. Sol receives maximal-deliberation instructions, while Astra uses think-here with native reasoning still enabled at low effort. Table 6 shows that prompt wording substantially changes Astra’s allocation between visible and native tokens. Consequently, differences in length and branching may partly reflect elicitation conditions. Common-prompt comparisons and budgets accounting for visible, native, and repeated input tokens are needed to support broader efficiency claims.

  3. Tree metrics may reflect expression granularity rather than reasoning efficiency. Appendix F constructs a separate sketch for each trace, defines depth by its largest step index, and excludes conclusion-only descriptions from work assignments. Thus, similar depth is not evidence of comparable reasoning depth, and omitted detail can itself reduce tree size. The episode-labeling pilot does not validate complete tree reconstruction or invariance to compact versus expanded wording.

  4. The evidence for greater semantic density is indirect. Zlib measures textual compressibility and is sensitive to length, formatting, and repeated strings. Similar activity-category proportions also do not establish equal amounts of correct inferential work. Appendix H’s observation that Astra’s units often have similar character lengths to other models supports sparse externalization, but does not establish generally denser reasoning. Matched-content controls or narrower claims are needed.

  5. The central efficiency claim does not distinguish authentic reasoning from memorization. The claim that "efficient reasoning is concise, dense, and directed" goes beyond the observed trace characteristics. Astra's shortest, nearly linear path in Figure 1 is consistent with efficient reasoning, but could also reflect prior exposure to the problem or its solution during training. Although the MATH set includes February 2026 HMMT problems, the paper does not report model-specific exposure checks or tests on newly constructed variants. Such controls would help distinguish adaptable reasoning from memorization or benchmark-specific overfitting.

  6. Cross-table consistency and statistical support require clarification. Astra’s Forced accuracy is 93.8% in Table 1 versus 95.0% for think-here in Table 6. Appendix A.2 describes zero native-token usage, whereas Table 6 reports a mean of 13.65 and a 93.75% zero-native rate. Different cohorts could explain this, but their mapping is unclear. Closed-model paired uncertainty and an explicit equivalence criterion are also needed to substantiate the abstract’s “matches” claim.

Questions for authors
  1. Proxy validity: Beyond the existing lexical and activity comparisons, how closely do native and forced traces agree on solution strategy, key intermediate dependencies, and error recovery on the same questions? Which internal-mechanism claims remain supported given the differences reported in Appendix K?

  2. Protocol and cost controls: Do the length and tree-structure rankings persist under common tool descriptions and comparable budgets? Please report visible and native output, repeated input processing, and tool-call counts separately, and distinguish observable token savings from total computational efficiency.

  3. Tree validity: Does rewriting the same verified solution into compact and expanded forms change the reconstructed tree substantially? Can the reported differences be validated against a shared reference derivation on matched questions where both models answer correctly?

  4. Semantic density: After matching text length and formatting, do the compressibility differences persist? What independent evidence establishes that shorter traces convey comparable correct inferential content, rather than merely omit more intermediate detail?

  5. Authentic reasoning: How does the proposed approach substantiate all three aspects of "concise, dense, and directed" reasoning while distinguishing them from memorization? In particular, what evidence addresses prior training exposure as an explanation for Astra's short, nearly linear path in Figure 1? Do the findings persist on newly constructed problems or controlled variants that change the required derivation, with correct intermediate steps adapting accordingly?

  6. Reproducibility and uncertainty: Please reconcile the samples and token definitions in Tables 1, 5, 6, and 8, including completed versus scheduled interactions and zero-native filtering. What decoding settings, output/tool-call limits, grading procedures, and repeated or paired evaluations underlie the closed-model comparisons?

3 comments

We thank the reviewer for the careful assessment and valuable feedback.


Response to Q1: Proxy Validity

This question conflates behavioral correspondence with mechanistic equivalence. Our objective is to redirect reasoning into the forced-reasoning tool, not to reconstruct inaccessible internal computation.

Our open-model validation examines performance, lexical similarity, functional composition, and reasoning dynamics (Appendices A and K). Appendix K reports both shared patterns and behavioral differences, as reasoning behavior varies across models.

For closed-source models, native CoT and internal architectures are inaccessible, and no ground-truth reasoning traces are available. For instance, Astra might employ a looped Transformer architecture or other architectural optimizations, but such mechanisms cannot be verified externally. We therefore focus strictly on observable differences in reasoning externalization through controlled cross-model behavioral comparisons, rather than attributing them to specific internal mechanisms.


Response to Q2: Protocol and Cost Controls

Please refer to Reviewer fNni, Weakness 3 for the unified tool description, and Reviewer 4U1Q, Q2 for length-matched comparisons (within 1.2×). Both support persistent differences in observable reasoning structure.


Response to Q3: Tree Validity

Our tree metrics characterize externalized reasoning structure, not invariant mathematical proof complexity. To assess expression granularity, we conducted a style-rewriting experiment on 28 HMMT problems correctly solved by both Astra and Sol.

Using one-shot prompting with a paired Astra/Sol example, Astra rewrote its traces toward Sol's style without access to the target Sol trace.

Mean metricOriginal AstraRewritten AstraOriginal Sol
Tree nodes17.9323.0731.75
Sketch depth11.0413.3210.89
Branch nodes2.503.295.57

Rewriting increased Astra's nodes and branching, confirming that expression granularity affects reconstruction. Yet rewritten Astra retained fewer nodes and branch nodes than Sol, although sketch depth did not follow the same ordering. Thus, differences in tree size and branching persist under more explicit expression, reflecting visible reasoning organization rather than internal computational complexity.


Response to Q4: Semantic Density

Please refer to Reviewer 4U1Q, Q2 for length-matched, correctly solved comparisons under identical tool descriptions, and Reviewer fNni, Weakness 2 for trace-completeness evidence.

Omitting explicit intermediate calculations does not necessarily mean omitting essential reasoning. Figures 8–9 and Table 2 illustrate how Astra consolidates multiple operations into fewer externalized units. Performing elementary calculations without explicitly expanding every step is itself a meaningful reasoning capability. Astra's 93.8% Forced MATH accuracy, despite substantially shorter traces, demonstrates that compact externalization can preserve strong task performance rather than sacrificing correctness for brevity.


Response to Q5: Authentic Reasoning and Memorization

The causal interventions in our response to Reviewer 4U1Q, Q3 show that Astra does not merely repeat supplied reasoning: it can reject misleading directions, correct arithmetic errors, and continue reasoning from corrupted intermediate steps. This provides evidence of active reasoning behavior rather than unconditional answer reproduction.

Potential memorization does not invalidate the observed differences in reasoning externalization. Our conclusions concern measurable behavioral patterns, not unverified internal mechanisms.


Response to Q6: Reproducibility and Uncertainty

For the 93.8% vs. 95% discrepancy between Tables 1 and 6, zero-native filtering, and three independent runs without correctness-based selection, please refer to Reviewer 4U1Q, Q4.

Decoding: We used provider defaults without modifying temperature, top-p, or other sampling parameters; several evaluated models do not support custom sampling settings.

Limits: No additional client-side output-token or forced-reasoning tool-call limits were imposed beyond provider constraints. Tool calls continued until a final answer (Appendix B).

Token accounting: Table 5 (Tool) sums tool-calling rounds, including tool framing but excluding the final-answer round. Table 8 (Forced OFO_F) counts whole-interaction completion tokens minus native reasoning tokens, including the final answer. We will clarify these definitions in the table captions.

Benchmark sampling and model configurations are documented in Appendices C and D.

I thank the authors for the detailed response. The clarifications on decoding, limits, and token definitions resolve my reproducibility questions, and the style-rewriting experiment is informative. However, my central concerns remain.

First, the abstract still makes mechanism-level claims (earlier trajectory selection, internal resolution of elementary steps). The response itself states that the paper concerns only observable reasoning externalization, so these claims should be revised to match.

Second, the rewriting experiment shows that wording alone increases Astra’s tree nodes by 29%, and rewritten Astra exceeds Sol in sketch depth. The tree metrics therefore depend largely on expression granularity and should be presented as such.

Third, the interventions show that Astra processes supplied steps. They do not address whether the short path in Figure 1 reflects prior exposure to the problem. Without a check on newly constructed variants, “efficient reasoning is concise, dense, and directed” should be restricted to the observed traces.

Fourth, the “matches” claim still lacks paired uncertainty across the three runs and a stated equivalence margin.

I therefore maintain my score of 2. I would raise it if the authors (1) provide revised wording for the abstract and the efficiency claim that limits both to observable externalization, and (2) report paired uncertainty for the closed-model Native vs. Forced comparison. A small check on variant problems would further strengthen the paper.

We thank the reviewer for the constructive follow-up. We will revise the abstract and efficiency claims to distinguish observable reasoning externalization from unobservable internal mechanisms.

Regarding rewriting, we believe the reviewer may have misunderstood the experiment's purpose. We deliberately rewrote Astra's traces to resemble Sol's more verbose style while preserving the reasoning. Thus, increased tree size is expected by design, not evidence against our findings. Even after rewriting, Astra retains 27.3% fewer tree nodes (23.07 vs. 31.75) and 41.0% fewer branch nodes (3.29 vs. 5.57).

Regarding paired uncertainty, we assume the reviewer is referring to question-level paired confidence intervals for Native–Forced accuracy differences. Both conditions were evaluated on the same benchmark questions. Our three additional independent runs were specifically designed to assess the robustness and reproducibility of Forced-Reasoning performance across repeated evaluations, rather than to conduct paired Native–Forced comparisons.

We appreciate the suggestions and will incorporate these clarifications into the revision.

Reviewer_4U1QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FORCED-REASONING, a protocol that elicits reasoning text from closed-source models through forced scratchpad tool calls. The authors validate the protocol using accuracy and text similarity on open-source models, then analyze four closed-source models.

The authors compare reasoning length, compression, episode distributions, and reasoning-tree structures, finding that Astra produces shorter, more compact traces with narrower trees. They also study reasoning-text transfer across models, observing larger performance drops when weaker models use Astra's traces. Finally, an HMMT experiment shows that forced tool calls substantially outperform regular step-by-step responses for Sol.

Critical strengths

1. A reproducible black-box protocol revealing interesting channel effects.

The protocol uses standard APIs without model-internal access. Open-source validation includes native CoT comparisons and 10 repeated samples. On HMMT, Sol's accuracy rises from 60.6% with regular responses to 84.8%–93.9% through tool calls, suggesting that output channels can substantially affect reasoning performance.

2. An interesting connection between reasoning-text transferability and model capability.

Section 4.4 fixes donor reasoning text while varying recipient capability. Weaker models perform worse with Astra's concise reasoning text, suggesting a potentially useful relationship between reasoning representation and recipient capability.

Critical weaknesses

1. The claim that Astra reasons more compactly, directly, and with less exploration is not adequately supported.

  • Length confounding: zlib compression ratio is strongly related to text length, while tree width and node count depend on episode counts. Astra actually has the highest EXPLORE proportion (12% versus Sonnet's 6%). Exploration/backtracking transitions and earlier correct-path selection are not measured. Mixing correct and incorrect trajectories may also confound the tree analysis.
  • Inconsistent settings: Sol uses maximal-deliberation, explicitly requesting all steps and alternatives; Astra uses think-here with residual native reasoning; Claude appears to use default settings. Sol's and Astra's descriptions were selected on the same MATH dataset used for analysis.
  • Weak local evidence: Section 4.2 presents only three selected excerpts. Figure 8 shows similar characters per episode, and Appendix H.1 acknowledges that Astra may simply externalize less reasoning rather than compress local expressions.

2. The faithfulness of tool-channel reasoning is insufficiently established.

  • Open-source validation: Negative controls are missing. Unigram ROUGE may reflect problem overlap, while 5-gram Jaccard is only 0.008–0.019. GLM-5.2's native–forced similarity is less than half its native–native similarity. Forced reasoning also reduces GLM's EXPLORE proportion from 12% to 3%, suggesting that the protocol changes reasoning characteristics.
  • Closed-source validation: Accuracy alone cannot establish reasoning equivalence. Tool-text/native-token length ratios vary widely (Astra 0.28, Sol 0.67, Sonnet 1.01, open-source models 1.4). Table 6 shows that stronger tool descriptions increase Astra's native reasoning while shortening tool text. This cannot distinguish compressed reasoning from generated summaries.
  • No causal validation: The authors do not intervene on replayed tool text through truncation or error injection to test whether subsequent reasoning depends on it.
Questions for authors
  1. Would the length, compression, and reasoning-tree conclusions hold under identical tool descriptions, or at least with Sol using the default description? Sol's and Astra's descriptions were selected on the same MATH dataset used for analysis.
  2. Do compression and reasoning-tree differences remain after controlling for text length? Could the authors report exploration/backtracking transition rates and restrict comparisons to problems all four models answer correctly?
  3. Could truncating or injecting errors into replayed tool text test whether final answers actually depend on it?
  4. What is Astra's accuracy at low effort without tools? Why does Appendix A.2 claim zero native reasoning tokens for all Forced runs when Table 6 reports only 93.75%? Were results filtered?
  5. When Sol and Opus are donors, how often do weaker recipients fail despite the correct answer appearing in the donor text? Could confidence intervals across repeated runs establish the stability of these results?
2 comments

We thank the reviewer for the detailed review and suggestions.


Response to Question 1: Unified Extraction Settings

Please refer to our response to Reviewer fNni's Weakness 3


Response to Question 2: Text Length and Reasoning Structure

We additionally compared five mathematics problems that Astra and Sol both solved correctly under identical forced-reasoning tool descriptions, selecting cases with similar output lengths to reduce length confounding.

The table reports mean values, with paired numbers denoting Astra / Sol.

Tool tokensZlib ratioTree nodesWidthDepthExplorationBacktracking
Mean706.6 / 770.0.511 / .49918.2 / 21.86.4 / 5.89.0 / 9.44.8 / 10.00.6 / 1.8

Exploration and backtracking are mean transition counts per trace, derived from the reasoning-activity labels used in our tree reconstruction (Appendix F).

Since Sol generally produces substantially longer traces than Astra, only a few cases met our matching criteria: both models answered correctly, with token counts differing by no more than 1.2×. Even in these length-matched cases, Astra exhibits fewer exploration and backtracking transitions, as well as fewer tree nodes.


Response to Question 3: Causal Interventions on Replayed Reasoning

We conducted three causal interventions on ten APEX problems originally solved correctly by Astra:

  1. Opening-trace truncation and misdirection: Retaining only the initial reasoning, introducing a misleading direction, and allowing continuation.
  2. Intermediate corruption: Corrupting a key calculation in a partially replayed trace and allowing continuation.
  3. Final-result corruption: Corrupting the final-result information in the complete trace and requiring an immediate answer-only response(further tool calls disabled).

Results. In the first condition, both corrupted and unmodified-prefix controls achieved 9/10 accuracy. In the second, both achieved 10/10. In the third, accuracy dropped from 10/10 to 1/10 after final-result corruption.

Observed behavior. Human inspection found additional forced-reasoning tool reasoning in all 20 opening and intermediate intervention trials, including explicit error correction and rejection of misleading strategies. For example, Astra rejected an incorrect parity argument ("Last parity suggestion plainly false N3 W N4 L discard") and corrected the injected arithmetic error ("earlier wrote7 error (13+3)/2 8").

Together, these causal interventions demonstrate that replayed forced-reasoning tool content can causally influence subsequent answers, while continued reasoning can detect and correct misleading information.


Response to Question 4: Native Reasoning Usage and Table Consistency

This concern stems from a misunderstanding of the experimental protocols. Table 1 strictly requires zero API-reported native reasoning usage, whereas Table 6 is a prompt ablation intentionally measuring native reasoning usage, including nonzero cases.

For Table 1, whenever native reasoning leakage occurs, we discard the entire trace regardless of correctness and rerun the same question until zero native usage is achieved. No questions are excluded, and no correctness-based selection is performed.

Thus, Table 6 reports 95% accuracy (76/80), versus 93.75% (75/80) in Table 1—a net difference of one correct answer. Table 6's 93.75% is the zero-native rate, not accuracy.

We additionally conducted three independent runs on the MATH benchmark:

Model / ConfigurationAccuracy (mean ± std.)
Astra think-here (effort low)93.8% ± 1.2%
GPT-5.6 Sol forced maxdelib (effort none)90.0% ± 1.2%

These results demonstrate consistent performance across runs, without correctness-based selection or exclusion of benchmark samples.


Response to Question 5: Recipient Performance Stability

We conducted four additional matched runs to assess the stability of cross-model reasoning reuse.

Donor → RecipientMean accuracy95% CI
Sol → Luna91.25%Identical across all runs
Sol → DeepSeek90.31%[89.32%, 91.31%]
Opus → Luna84.06%[83.07%, 85.06%]
Opus → DeepSeek84.69%[83.69%, 85.68%]

Sol → Luna achieved exactly 91.25% accuracy in every run. The other configurations also exhibited narrow confidence intervals, supporting the run-to-run stability of the observed recipient performance.

Thank you for the detailed rebuttal. The experiment using the same think-here tool description on all 80 MATH problems largely addresses my concern about inconsistent tool settings. The causal intervention experiments also provide stronger evidence for the authors' claims. However, some concerns remain:

  1. The length-matched comparison includes only five problems, which is a very small sample. In this subset, Astra's reasoning trees are actually slightly wider than Sol's. This suggests that the differences in tree width may be influenced by output length, task performance, or sample selection, rather than simply reflecting differences in the models' reasoning strategies.

  2. The authors have not reported Astra's accuracy at low effort without tools, or the proportion of cases where recipients give incorrect answers despite the correct answer already appearing in the reasoning traces provided by Sol or Opus.

  3. The causal interventions show that replayed tool text can influence subsequent answers, but they do not provide clear evidence that the extracted traces faithfully reflect native CoT.

Overall, the additional experiments strengthen the empirical support for the paper's findings. However, the central claims about the underlying reasoning mechanisms remain insufficiently supported. I therefore maintain my original rating.

Reviewer_mjAoOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a CoT extraction method targeting closed-source LLMs. They first apply the method to open-source models to validate its performance, and then apply it to major proprietary models to reveal their reasoning structures. The main contributions of the paper lie in the proposed method for extracting useful CoT from closed-source models and the behavioral insights it provides.

Critical strengths
  1. The performance comparison on open-source models supports the claims and arguments of the paper from both quantitative (numeric results) and qualitative (lexical similarity) perspectives.

  2. The experiments analyzing the extracted traces are thorough. The insights regarding how extracted CoT can boost or degrade student LLMs are particularly useful.

Critical weaknesses
  1. As discussed in the Appendix, the proposed method relies heavily on the models' post-training strategies and inherent generalization capacities. The method extracts CoT from tool-calling arguments, which are typically compressed and not explicitly trained for reasoning. Therefore, the method depends on the LLM's ability to generalize its reasoning capacity to an out-of-distribution format. I suspect this will not work well for smaller models, which aligns with the paper's experiments where the successfully extracted traces are all from frontier-class models, where luna type models are missed in the experiment.

  2. As noted in the paper, some models (like GPT-6 Astra) do not allow native reasoning to be fully turned off. On such models, the proposed method is inherently confounded. Even though Astra's reasoning effort is set to "low," it is highly likely still reasoning internally. This internal reasoning probably contributes to its strong performance, its more linear reasoning trees, and the observation that it skips minor elementary steps in the externalized trace.

Questions for authors
  1. Can the authors provide more evidence of whether the proposed method works on smaller-scale models, such as the Luna series or other smaller open-source models?

  2. While the comparisons of reasoning traces across closed models are insightful, I am unsure how the outcomes will practically benefit the broader community. Publishing these extracted traces at scale could face legal or Terms of Service issues. Do the authors primarily expect the proposed extraction method itself to be adopted by the community for future behavioral research?

1 comment

We thank the reviewer for the thoughtful comments.


Response to Weakness 1: Applicability to Smaller Models

We additionally evaluated Luna and Nano, both of which successfully generated reasoning through the forced-reasoning tool with zero API-reported native reasoning tokens. On the MATH benchmark, GPT-5.6 Luna achieved same accuracy under both conditions:

ConfigurationAccuracyMean tokens
Native-medium39/80 (48.75%)3,544
Forced-reasoning tool39/80 (48.75%)3,150

These results demonstrate that Luna can also exhibit the same forced-reasoning behavior; this further suggests that the proposed method is not limited to frontier-class models.


Response to Weakness 2: Native Reasoning in Astra

The configured reasoning effort and actual native reasoning usage are distinct. Although Astra requires at least low effort, all retained main-experiment interactions have zero API-reported native reasoning tokens (Appendix A.2). Any interaction with nonzero usage is rerun on the same question, regardless of answer correctness. Thus, the reported results are not confounded by measured native reasoning token usage. In other words, all analyzed visible reasoning traces are free from contamination by native reasoning tokens.


Response to Question 2: Practical Value and Transparency

Our primary goal is not to provide a reproducible extraction recipe, which is why we only include illustrative pseudocode. Rather, our goal is to advance the academic community's understanding of frontier reasoning models, where transparency is essential for interpreting their capabilities and evaluating their reliability.

For example, on additional benchmarks not reported in the paper, we observed models explicitly stating in their reasoning traces that they recognized the problem and then directly recalling its answer, rather than deriving a solution. Such behavior is invisible from final-answer accuracy alone and raises important questions about memorization and benchmark validity. This illustrates why greater transparency into reasoning behavior is crucial for the scientific evaluation of frontier models, beyond simply reporting benchmark scores.

We also proactively disclosed our methodology, implementation, and code to OpenAI and Anthropic, together with potential mitigations.

Reviewer_fNniOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses forced tool calls to get models to write their intermediate reasoning into a text argument of a scratchpad tool. The authors first validate this approach on open-source models by comparing task accuracy and text structure against their accessible native CoT. They then analyze the output length, redundancy, reasoning behavior, and branching of 4 closed-source models. GPT-6-Astra produces shorter traces with fewer branches, but some recipient models cannot fully benefit from these traces when given them as context. The paper provides behavioral observations of hacked CoT utilization, but cannot confirm that these reasoning traces reflect the closed-source models’ actual internal reasoning.

Critical strengths
  • The research question is clear and has some novelty: beyond final-answer accuracy, the paper examines how models express and organize their reasoning.
  • The experimental analysis is fairly comprehensive. Open-source model validation, prompt ablations, checks of native reasoning-token usage, and cross-model reuse experiments complement each other.
  • Different recipients benefit differently from the same donor’s reasoning. Some even produce incorrect answers when the correct answer is already present in the input. This is an interesting finding that deserves further study.
Critical weaknesses
  • The zlib metric provides limited evidence about useful reasoning information density. The authors use the zlib compression ratio as a rough measure of textual redundancy. However, text being harder to compress does not necessarily mean that each token contains more useful mathematical reasoning, nor does it directly measure semantic repetition. The connection between this metric and useful reasoning information density therefore still needs direct validation.
  • The reasoning traces given to recipients may be incomplete. The protocol only forces the donor model to call the forced-reasoning tool in the first round. In later rounds, the model can choose whether to call the tool again or produce its final response. If that final response contains key derivations that were not written into the tool’s arguments, those steps may be missing from the recipient’s input. The paper does not sufficiently examine this possibility. Some recipients’ lower performance may therefore reflect missing steps in their input, rather than only overly concise reasoning or limitations in their capabilities.
  • Cross-model comparisons are affected by differences in generation settings. For example, Sol is asked to explicitly write out intermediate steps, alternative approaches, and checks, while Astra uses a different tool description and keeps native reasoning enabled at low effort. The authors’ own prompt ablations show that these settings can substantially change the length of the visible reasoning. It is therefore difficult to separate differences between the models themselves from differences caused by the generation settings. This weakens the interpretation that shorter traces and fewer branches directly indicate more efficient reasoning by the model itself.
Questions for authors

None.

1 comment

We thank the reviewer for the constructive feedback.


Response to Weakness 1: Zlib Compressibility and Semantic Information

We agree that zlib compressibility is sensitive to text length and formatting and cannot fully capture semantic information. However, Figures 8 and 9 show that Astra has comparable average characters per annotated reasoning unit but substantially fewer total characters. Our manual inspection, including the examples in Table 2, further suggests that Astra often consolidates multiple reasoning operations expressed as separate units by other models into a single unit.

Furthermore, the PuzzleWorld case study in Appendix G illustrates how Astra skips explicit intermediate deductions in multi-step logical reasoning, directly expressing key conclusions without spelling out every intermediate implication. We will provide additional matched cross-model cases to further substantiate these observations.


Response to Weakness 2: Completeness of Transplanted Traces

Although only the initial tool selection is forced, all subsequent reasoning arguments from the forced-reasoning tool are included in the transplanted trace (Appendix B). Only the final response is excluded.

During our original reasoning-activity annotation, we manually inspected 20% of the mathematical traces and observed no cases where the final response introduced additional derivations absent from the forced-reasoning tool output. The final response summarized the existing reasoning and stated the final answer.

Moreover, Appendix I shows that recipient failures frequently occur even when the correct answer is explicitly present in the transplanted trace, suggesting that missing answers are not the primary explanation for the performance gap.


Response to Weakness 3: Cross-Model Generation Settings

We agree that prompts can affect visible reasoning. We additionally conducted a matched-tool-description experiment on all 80 MATH questions, using exactly the same think-here tool description for Astra and Sol.

Model / configurationAccuracyForced-reasoning tokensTree widthTree depthTree nodes
Astra think-here (low effort)93.75%1,282.15 / 7.3011 / 14.9127 / 41.01
GPT-5.6 Sol think-here (non-reasoning)73.75%1,883.88 / 11.4110 / 10.0034.5 / 41.44

Under identical tool descriptions, Astra retains higher accuracy, shorter forced-reasoning traces, and narrower trees. Its median node count is also lower, although mean node counts are nearly equal and its tree depth is greater.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.