- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a Coordinated Representation Learning (CRL) framework for systematic reasoning in large language models, aiming to address the problem of existing reasoning enhancement mechanisms being independent and lacking effective coordination. Specifically, this framework integrates granularity-invariant thought chain supervision, multi-teacher latent space knowledge distillation, difficulty-aware curriculum learning, and structured subspace parameter adaptation, optimizing the model's reasoning representation through a unified objective. The authors conducted experiments on seven benchmarks, including mathematical, logical, and code reasoning, covering models with parameter scales from 0.3B to 14B. Experimental results show that CRL outperforms SFT and other independent training baselines in most settings.
Critical strengths
- The latent space distillation design has certain value. Compared to knowledge distillation that only focuses on output distribution, this method further utilizes intermediate layer representations from multiple teacher models to transfer inference knowledge.
- The experimental coverage is relatively broad. The paper evaluates the method on seven inference benchmarks and models of multiple parameter scales, covering different tasks such as mathematical, logical, and code generation.
- The manuscript structure is clear.
Critical weaknesses
- The core synergistic effect lacks sufficient validation. The paper claims that the benefits of coordinated optimization exceed the sum of gains from independent mechanisms, but lacks a systematic analysis of interaction effects, which is insufficient to prove the superadditive synergistic relationship.
- The rationale for the consistency of long and short CoT representations is insufficiently validated. Inference trajectories of different granularities may contain different key intermediate information, and directly constraining latent space consistency may weaken the fine-grained information required for complex inference.
- The difficulty measurement of curriculum learning suffers from scaling issues. Eq. (6) directly combining confidence, normalized CoT length, and undefined normalized representation distance may lead to an imbalance in the contributions of different difficulty terms.
- The computational efficiency analysis is insufficient. Although the paper reports training time and inference latency, it does not fully account for the upfront costs of multi-teacher inference, CoT generation, and representation caching, and the actual efficiency after Adapter merging lacks sufficient validation.
- The fairness of comparisons between different training baselines is insufficient. CRL utilizes multi-teacher distillation, long and short CoT, and Adapter optimization simultaneously, but it is unclear whether each baseline uses matched data, teacher information, and training budgets.
Questions for authors
- It is recommended to add complete component combination and interaction effect experiments under the same training budget to verify whether coordinated optimization truly brings superadditive benefits.
- Further verify whether long and short CoT alignment preserves key inference steps.
- It is recommended to explain the normalization methods for the three difficulty metrics and provide sensitivity analyses of the difficulty score distribution, number of neighbors, and weights of each metric.
- It is recommended to supplement the end-to-end computational costs, including teacher inference, CoT construction, and representation caching.
- Could the authors clarify the training data, teacher information, token generation budget, and computational resources used for each baseline, and add performance comparisons under strict matching conditions?
- The main paper exceeds the 9-page limit; please note!
First, I would like to express my sincere gratitude to the reviewers for pointing out areas for improvement.
Q1
The specific experiments are as follows:
We constructed a 2⁴ factorial design (four-factor factorial), with each factor taking the values “present/absent,” resulting in a total of 16 training conditions. We trained the model on Qwen2.5-Math-7B using the same set of seeds. The specific results are as follows:
Table: Factorial interaction analysis on Qwen2.5-Math-7B (mean over 5 seeds). Δadd is the sum of the relative gains of individual components compared to SFT; ΔCRL is the observed combined gain; Δsyn=ΔCRL−Δadd.
And report a key summary table:
Table: Same-budget full-combination summary (Qwen2.5-Math-7B, mean over 5 seeds). Δsyn is defined as in the table above.
Q2
We conduct retention probes on two types of steps in long CoT chains:
(i) answer-critical spans, Steps whose removal causes the final answer to change; (ii) intermediate results unique to long chains, intermediate results that appear only in the reasoning process of long chains.
Table: Comparison on Qwen2.5-Math-7B
The experiments revealed that the NLDD values playing a critical role are concentrated in the first 70–85% of the chain; After alignment, the key steps and unique intermediate quantities in long chains that truly influence the answer are largely preserved, with only minor differences (-0.4%).
we preliminarily conclude that alignment preserves the critical inference steps.
Q3
Thank you for pointing out the correction. Here is a brief summary about normalization methods:
After mapping all three metrics to the range [0, 1], they are linearly weighted: the confidence term 1−ci takes values directly in [0, 1]; the length term ∥ri∥/maxj∥rj∥ is normalized based on the maximum trajectory length within the batch; the variance term σi is first min-max normalized based on the batch’s k-NN average distance, then included in the weighting.
The sensitivity analysis is as follows:
Table: Difficulty-weight sensitivity (Qwen2.5-Math-7B, MATH500 accuracy). Default values are in bold.
Q4
Thank you for the suggestions. I have summarized them as follows:
Table: End-to-end cost breakdown for CRL training pipeline (Qwen2.5-Math-7B setting). Teacher-side costs are one-time and amortized across student training runs.
Q5
Clarification Summary:
All baselines share the same student checkpoint, the same training/validation split, the same teacher ensemble, self-consistency, retention, the same CoT construction protocol, and the same LoRA trainable parameter budget. Computing resources are 8×A100 80GB. The only aspect that is not strictly matched is the number of training steps: CRL’s wall-clock training time is 12% longer than that of full-parameter SFT (13.8 vs. 12.3 h), but the number of training steps, batch size, and number of epochs are exactly the same as SFT’s.
Q6
Thank you for pointing this out.
End
Thank you again for your review.