acceptodds
Under review as a conference paper at ICLR 2027

Offline Guidance, Online Reasoning: Reusing LLM Feedback for Small Language Models

Abstract

Large language models (LLMs) offer strong reasoning capabilities but are often costly to access through commercial APIs, while small language models (SLMs) are easier to deploy locally yet remain weaker in reasoning. This capability–deployment gap has motivated LLM–SLM collaboration, which aims to improve SLM reasoning using LLM capabilities while preserving the deployment advantages of SLMs. Existing approaches mainly follow two paradigms. Knowledge distillation uses LLM-generated answers and reasoning trajectories to train SLMs offline, but requires parameter updates and additional training. Alternatively, online collaboration routes difficult problems to an LLM or leverages LLM-generated guidance and corrections when an SLM encounters difficulties. Although effective, online collaboration requires repeated LLM access. Moreover, the guidance produced for a particular problem is discarded after inference and cannot benefit subsequent problems involving similar reasoning states. In the paper, we focus on a more constrained setting in which the LLM is accessed only offline, the SLM parameters remain fixed, and online inference is performed solely by the SLM. To this end, we propose Reusable Latent Correction (RLC), which converts one-off natural-language guidance from a black-box LLM into persistent corrective experiences in the hidden space of an SLM. RLC stores these experiences in an external bank and retrieves them according to the SLM’s current reasoning state, enabling the SLM to reuse LLM-derived corrections during inference without any online LLM calls. Experiments across multiple reasoning benchmarks and SLM scales show that RLC consistently improves SLM reasoning without parameter updates or online LLM calls.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Model Collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cfDqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Reusable Latent Correction (RLC) for a setting in which a large model is available during preparation but not at test time. During offline construction, the large model gives feedback on uncertain steps produced by a frozen small model. RLC turns the resulting changes in the small model’s hidden states into entries in a key–value bank. At test time, a router decides whether to activate the bank; if activated, the small model retrieves corrections using its current reasoning state and applies them during generation. The paper reports accuracy gains on five benchmark families with three small-model sizes. Average inference latency is reported to be 3.1–4.2% above native inference.

Critical strengths
  1. The paper addresses a practical constraint: teacher feedback can be collected offline, while the deployed small model must run without teacher calls or parameter updates. The proposed bank gives a concrete way to reuse that feedback across problems.
  2. The evaluation goes beyond a native-model comparison. The prefill-only and decode-only ablations, the always-on bank results, and the mean-value control help identify which parts of the system contribute to performance. Several of the reported accuracy gains are large enough to merit further investigation.
Critical weaknesses
  1. The results do not yet separate the value of the stored corrections from that of the trained router. In Appendix Table 6, always enabling the bank improves average accuracy over native inference by 3.93, 3.52, and 1.93 points for the 3B, 7B, and 14B models. Adding the router yields a further 5.79, 4.45, and 4.00 points. The mean-value control is useful evidence for state-matched retrieval, but the paper gives too little information about router training-set sizes, class sampling, and validation to assess how reliably the routing gains would generalize. A control that keeps the router and intervention schedule fixed while changing how corrections are matched would make the main claim more convincing.
  2. Some reporting issues make the quantitative conclusions difficult to assess. The paper does not give the numerical uncertainty threshold, final bank sizes, or a complete account of validation and hyperparameter selection. It also reports no uncertainty intervals or paired tests, which matters particularly for AMC24 (45 problems) and AIME (30 problems). Moreover, native-model results differ between Tables 1 and 5 without an explanation: 32B MATH500 accuracy is 83.20 in Table 1 but 82.80 in Table 5, while 14B GPQA output length is 421.87 versus 634.18. These discrepancies affect the interpretation of the cross-scale comparisons. Finally, the 3B experiments use AIME25, whereas the larger-model experiments use AIME24. Averages across those groups should therefore not be treated as directly comparable.
Questions for authors
  1. What uncertainty threshold and final bank size were used for each model? How many examples trained each router, what were the class counts, and which data were used to select the dataset-specific intervention settings?
  2. What accounts for the native-model differences between Tables 1 and 5? Were the results obtained from different runs, prompts, or decoding configurations?
  3. Could you provide paired native, always-on bank, and routed-bank outcomes with uncertainty estimates? A comparison that holds the router fixed but randomly reassigns corrections to retrieved states would also help establish the contribution of the stored key–value associations.
6 comments
AuthorsOpenReview ↗

Response to Reviewer cfDq(1/4)

We are grateful for your detailed feedback on the experimental analysis and reporting.Your suggestions have guided us in further examining the contributions of the Router and correction vectors and clarifying the evaluation protocol.We respond below with additional control experiments,methodological details,and corrected results.

W1.We do not yet separate the value of the stored corrections from that of the trained router.

Thank you for suggesting that we further distinguish the contributions of the Router and correction vectors.In the original paper,we analyzed the role of the Router through ablation experiments and examined the importance of state matching through the Mean Value control.Below,we first clarify the existing results and then present the newly added Shuffled control experiment to further verify the role of correction vectors.

(1)To verify the effectiveness of the Router,we conducted an ablation experiment,as reported in Table 6 of Appendix C.3.Always enabling the Bank without the Router improves average accuracy over Native by 3.93,3.52,and 1.93 percentage points for the 3B,7B,and 14B models,respectively.Adding the Router yields further improvements of 5.79,4.45,and 4.00 percentage points,demonstrating its effectiveness.

(2)We agree that the effects of the correction vectors have not yet been fully isolated.In the original paper,we included a Mean Value control to examine the importance of state-matched corrections.To further verify the effectiveness of the correction vectors,we added a Shuffled experiment:while keeping the Router and retrieval rules fixed,we randomly shuffled the key-value correspondences in the Bank and reran the experiments using this shuffled Bank.The results are shown below.

Table R2.Results after shuffling the pairings in the Bank. Each cell reports accuracy(%)/ average output tokens.Averages assign equal weight to the 5 datasets.

ModelMethodMATH500GSM8KGPQAAMC24AIMEAverage
3BRLC73.00 / 576.9189.99 / 272.6840.91 / 690.9224.44 / 992.7810.00 / 921.9347.67 / 691.04
Mean Value70.00 / 574.7386.28 / 272.2936.36 / 723.9720.00 / 1003.7810.00 / 963.7744.53 / 707.71
Shuffled33.60 / 565.4154.28 / 272.7223.23 / 692.492.22 / 1024.713.33 / 960.5323.33 / 703.17
7BRLC79.80 / 564.7094.09 / 279.0745.45 / 654.4437.78 / 1100.3820.00 / 1300.5355.42 / 779.82
Mean Value76.80 / 568.9193.71 / 279.4840.40 / 652.5735.56 / 906.0710.00 / 1335.2751.29 / 748.46
Shuffled41.00 / 585.9064.22 / 279.2923.23 / 653.988.89 / 978.163.33 / 1142.2028.13 / 727.91
14BRLC83.00 / 583.8795.68 / 280.4552.02 / 467.1237.78 / 864.5820.00 / 1120.0757.70 / 663.22
Mean Value78.60 / 592.0695.30 / 280.3448.51 / 470.3133.33 / 941.6910.00 / 1105.8353.15 / 678.05
Shuffled42.00 / 578.7481.80 / 280.3330.81 / 469.4311.11 / 977.583.33 / 1056.8033.81 / 672.58

As shown,randomly shuffling the key-value correspondences in the Bank substantially reduces accuracy even when the Router and retrieval conditions remain unchanged.This is because the disrupted correspondences result in randomly mismatched interventions on Bank-routed problems,preventing them from being solved correctly.Together with the Mean Value experiment,this also demonstrates that correction vectors can indeed provide step-level corrections matched to the current reasoning state for problems that Native cannot solve correctly.

AuthorsOpenReview ↗

Response to Reviewer cfDq(2/4)

W2.The experimental settings and validation procedures are insufficiently described.

(1) Uncertainty threshold: To determine the uncertainty threshold(JSD threshold),we first sample 5% of the data from each batch and calculate their average step-level JSD,which is then used as the threshold.

(2) Final experience bank size: The actual sizes of the Banks constructed separately for the 3 models are shown below.As model capability increases,the corresponding correction experiences,i.e.,the number of experience entries,decrease.However,memory usage per entry still increases because the vector dimension grows.

BankNumber of experience entriesTensor memory(MiB)KiB per experience entry
3B4,521105.9624
7B4,000164.0642
14B3,549207.9560

(3) Hyperparameter selection and validation procedure: For validation and parameter selection on more difficult test sets,such as AIME and AMC,we randomly sample 10% of the level 4 and 5 problems from the MATH training set for tuning.For GSM8K,we sample 10% of its training set for tuning.For MATH500,we likewise sample 10% of the MATH training set for tuning.For GPQA,we tune the hyperparameters on the approximately 250 problems remaining in GPQA Main after excluding the test problems.Regarding the specific tuning procedure,a coarse search of the similarity threshold indicated better performance above 0.9.We first try top-k values of 1,2,and 3 while keeping the other parameters at their default values.We then sweep the similarity threshold over 0.95±0.020.95 \pm 0.02 in increments of 0.01.The process-vector injection strength is swept over 0.5±0.20.5 \pm 0.2 in increments of 0.1,and the result-vector injection strength over 0.25±0.10.25 \pm 0.1 in increments of 0.05.Because the result vector causes larger fluctuations,we use a slightly smaller injection strength and a smaller step size.We report paired results for Native,always enabling the Bank,and using the Bank with the Router on the same set of samples,together with uncertainty estimates.

AuthorsOpenReview ↗

Response to Reviewer cfDq(3/4)

(4) Confidence intervals and paired statistics: The tables below report the accuracy of the Native,Bank and Router as percentages."±\pm"denotes the sample standard deviation(SD)across 5 random seeds, measured in percentage points.Table 6 of the paper already reports the proportions of problems that remain on Native and those that activate the Bank under the Router for different datasets."Only X correct / Only Y correct"gives the number of problems answered correctly only by X and only by Y,respectively,showing the detailed paired outcomes for problems on which the Bank and Native branches disagree.N,B,and R denote Native,always enabling the Bank,and selecting a branch for each problem using the Router,respectively.

Method(3B)/ Paired comparisonMATH500GSM8KGPQA-DiamondAMC24AIME24
Native(N)67.00 ± 0.6284.69 ± 0.3030.30 ± 0.804.44 ± 1.573.33 ± 2.36
Bank(B)65.00 ± 0.6883.70 ± 0.3231.82 ± 0.9422.22 ± 1.573.33 ± 2.36
Router(R)73.00 ± 0.6089.99 ± 0.2740.91 ± 0.5124.44 ± 1.573.33 ± 2.36
Only B correct / Only N correct39 / 4976 / 8930 / 279 / 10 / 0
Only R correct / Only B correct42 / 285 / 225 / 71 / 00 / 0
Only R correct / Only N correct37 / 774 / 423 / 29 / 00 / 0
Method(7B)/ Paired comparisonMATH500GSM8KGPQA-DiamondAMC24AIME24
Native74.20 ± 0.4290.83 ± 0.2138.89 ± 1.0120.00 ± 1.5713.33 ± 2.36
Bank74.00 ± 0.5590.75 ± 0.2537.88 ± 1.0735.56 ± 2.2216.67 ± 2.36
Router79.80 ± 0.6394.09 ± 0.1645.45 ± 0.6237.78 ± 1.5720.00 ± 3.33
Only B correct / Only N correct29 / 3043 / 4428 / 308 / 13 / 2
Only R correct / Only B correct30 / 144 / 021 / 61 / 01 / 0
Only R correct / Only N correct28 / 043 / 022 / 98 / 03 / 1
Method(14B)/ Paired comparisonMATH500GSM8KGPQA-DiamondAMC24AIME24
Native77.00 ± 0.4594.47 ± 0.1742.93 ± 1.1331.11 ± 1.5713.33 ± 2.36
Bank79.40 ± 0.5193.25 ± 0.2341.41 ± 0.7137.78 ± 2.2216.67 ± 2.36
Router83.00 ± 0.4995.68 ± 0.1452.02 ± 0.8737.78 ± 1.5720.00 ± 2.36
Only B correct / Only N correct33 / 2119 / 3522 / 254 / 12 / 1
Only R correct / Only B correct21 / 335 / 325 / 41 / 11 / 0
Only R correct / Only N correct30 / 016 / 018 / 03 / 02 / 0
Reviewer_2CkVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper seeks to examine collaboration between large language models (LLMs) and small language models (SLMs) to solve reasoning problems. Different from existing collaboration methods that either require additional training or repeated access to the LLM, the authors propose a technique they term Reusable Latent Correction (RLC), where SLM parameters remain fixed and the LLM is only accessed offline. Through experiments, it is shown that the use of RLC improves reasoning capabilities of SLMs without needing online LLM access or update of parameters.

Critical strengths
  1. The main problem that the problem seeks to solve is well-motivated, and the paper clearly identifies possible drawbacks in two existing mechanisms used to enable collaboration between LLMs and SLMs. The formalization of the problem and the proposed RLC in Sections 3, 4 is also written clearly and in an easy to follow manner.

  2. Experiments presented clearly demonstrate the merits of the proposed RLC in terms of significant performance improvement in reasoning capabilities when SLMs are equipped with RLC. Please also see Point 3. of Weaknesses, below.

Critical weaknesses
  1. Is it the case that the effectiveness of RLC will depend on how frequently the experience bank will need to be updated and also the size (i.e., memory reqruirements) of the experience bank? In Fig. 1 (a) for example, the additional costs incurred in retraining will depend on the frequency with which retraining is required. Is there a reference that the authors could point to which alludes to and compares the effective costs and frequency of retraining vs updating the experience bank? I believe that the approach in Fig. 1 (c) will still be cheaper, all things considered, but it will be nice if the claim is supported by solid evidence.

  2. The computational complexity of accessing entries in the experience bank has not been clearly documented in the paper. Also, some more detail on how frequently, or in what fraction of instances, the experience bank was indeed accessed (based on Algorithm 1) will be helpful. It is not clear if Fig. 3 and Lines 405-417 describe this- if this is the case, what is the average computed over?

  3. A more critical weakness is that the experiments as stated in Lines 347-353 are limited to using models- both LLMs and SLMs- from only one provider (i.e., Qwen). A more objective assessment of the performance of RLC can be achieved through the evaluation of models from multiple providers.

Questions for authors
  1. Please see Weaknesses identified, above.

  2. In Lines 41-42, the paper claims that '...(SLMs) with 3B-14B parameters offer flexible deployment, low inference costs, and access to their internal states.' While the first two aspects are almost always true, it is always the case that SLMs provide access to their internal states by default, or is this an assumption that is being made? A clarification would be helpful.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2CkV(1/2)

We appreciate the time and effort you devoted to reviewing our work. Your questions prompted us to provide a more detailed account of Bank memory usage, construction costs, and retrieval overhead, together with evaluations across different model providers. Below, we present these analyses and clarify the deployment assumptions underlying RLC.

W1.We do not sufficiently quantify the memory usage and construction cost of the Bank.

The actual sizes of the Banks constructed separately for the 3 models are shown below.As model capability increases,the corresponding correction experiences,i.e.,the number of experience entries,decrease.However,memory usage per entry still increases because the vector dimension grows.

BankNumber of experience entriesTensor memory(MiB)KiB per experience entry
3B4,521105.9624
7B4,000164.0642
14B3,549207.9560

Conventional supervised fine-tuning(SFT)uses a shared dataset that can be applied to multiple models,but it still requires separate training for each model,involving relatively expensive GPU resources and substantial costs.Our approach also requires constructing a separate Bank for each model,but this process is training-free and requires only forward passes.Once constructed,the Bank can be used directly during online inference,where model parameters remain frozen and only relevant correction vectors are retrieved in real time.Therefore,for each individual model,constructing a Bank costs less than training the model with SFT.Specifically,the table below compares our Bank-based approach with SFT.

MethodDataOffline preparationOnline applicationTotal cost
SFTShared datasetFine-tuning Model 1Frozen Model 1High
Fine-tuning Model 2Frozen Model 2
Fine-tuning Model 3Frozen Model 3
⋮\vdots⋮\vdots
Fine-tuning Model nFrozen Model n
RLC(Ours)Shared datasetConstruct Bank 1Frozen Model 1 + Bank 1Low
Construct Bank 2Frozen Model 2 + Bank 2
Construct Bank 3Frozen Model 3 + Bank 3
⋮\vdots⋮\vdots
Construct Bank nFrozen Model n + Bank n

Below,we compare the costs of SFT and RLC based on our experimental statistics,using 2,400 problems selected from MATH and GSM8K.We use Qwen2.5-7B-Instruct as an example.The results are shown in the following table.

MethodGPUshour/GPUGPU cost($)API cost($)Total cost($)
LoRA SFT(5 epochs)4 ×\times A800 80 GB5.518.39-18.39
RLC6 ×\times RTX 409035.077.212.27

As shown,for the 7B model,the total cost of RLC is $6.12 lower than the training cost of SFT.

AuthorsOpenReview ↗

Response to Reviewer 2CkV(2/2)

W2.We do not sufficiently report Bank access frequency or analyze its retrieval complexity.

The Bank contains NN retrieval keys of dimension dd which are normalized when loaded.For each query,the normalized query vector is multiplied by all NN keys using dot products to obtain cosine similarities.The default experimental setting is K=3K = 3,and the dominant cost of a single retrieval is O(Nd)O(Nd).If a problem initiates SS queries,its total retrieval cost is O(SNd)O(SNd).

Retrieval stepTime complexity per query
Scan all NN keys and compute cosine similaritiesO(Nd)O(Nd)
Select top-K\mathit{top}\text{-}K from NN similarity scoresO(Nlog⁡(K+1))O(N \log(K+1))
Compute a weighted sum of correction vectors from KK experiencesO(Kd)O(Kd)

Figure 3 and Table 8 in Appendix C.5 report the average number of interventions triggered for Bank-intervened problems in each dataset.We first calculate the average number of interventions for RLC 3B,7B,and 14B separately on each dataset,and then take an equally weighted average of these model-specific averages for each dataset.The values for MATH500,GSM8K,GPQA,AMC24,and AIME are 6.17,4.65,1.06,10.61,and 8.53 interventions,respectively.Intervention frequency varies by task.Competition mathematics tasks,AMC24 and AIME,require the most frequent interventions,GPQA requires the fewest,and MATH500 and GSM8K fall in between.This result indicates that tasks involving longer and more complex reasoning processes tend to require more sustained correction.

W3.We do not sufficiently evaluate RLC on student models from different providers.

Due to time and computational resource constraints,we conducted experiments on student models from two other providers,Llama3.2-3B-Instruct and Gemma3-12B-Instruct,using 3 datasets.We also evaluated Qwen3.5-9B.These experiments cover models from different providers and with different parameter counts to thoroughly verify the effectiveness and model-agnostic nature of RLC.The results are shown below:

Each cell reports accuracy(%)/ average output tokens.The Average column assigns equal weight to the 3 datasets.

ModelMethodMATH500AMC24GPQA-DiamondAverage
Llama3.2-3BNative45.20 / 715.866.67 / 1330.0222.22 / 1056.1324.70 / 1034.00
Llama3.2-3BRLC52.80 / 799.6417.78 / 1412.9633.33 / 1083.4634.64 / 1098.68
Qwen3.5-9BNative93.20 / 1321.0868.89 / 2349.4256.57 / 2588.4572.88 / 2086.32
Qwen3.5-9BRLC95.40 / 1260.0275.56 / 2479.8463.13 / 2554.5878.03 / 2098.15
Gemma3-12BNative80.80 / 833.0848.89 / 1544.4236.87 / 878.7155.52 / 1085.40
Gemma3-12BRLC85.40 / 758.4657.78 / 1475.3647.98 / 909.4763.72 / 1047.76

Compared with Native,RLC improves the average accuracy across the 3 datasets by 9.94,5.14,and 8.20 percentage points on Llama3.2-3B,Qwen3.5-9B,and Gemma3-12B,respectively.This clearly demonstrates the effectiveness and model-agnostic nature of RLC.

Q1.Please address the major weaknesses above.

Please see the responses in the Weaknesses section above.

Q2.Is it assumed that SLMs always provide access to their internal states?

This is our assumption: local deployment of SLMs is common and allows access to their internal states.Specifically,In our experiments,all student models are deployed locally.We can load their weights and control the inference process,allowing us to read and intervene in the hidden states of selected layers.Student models can be called through APIs for inference.However,standard APIs typically only receive inputs and return generated outputs,while model parameters and layer hidden states remain with the service provider.

Reviewer_2CkVOpenReview ↗

Thank You

Thank you for your response. I will retain my current score, since the magnitude of the proposed updates to the paper in response to all reviewers' comments seems to be quite significant.

Reviewer_jRKeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study how to improve the math reasoning performance of small language models in a constrained setting, where no training or stronger LLMs are accessible. To this end, they propose Reusable Latent Correction (RLC), a method consisted of offline experience bank construction and online inference state intervention. Specifically, RLC uses layer prediction disagreement to identify uncertain steps, and appends a stronger model’s guidance for the model to regenerate that step. For the effective corrections, hidden states differences are recorded and stored as key-value pairs to build the offline experiment bank. During the inference stage, RLC employs a router to filter the problems that are likely to require such interventions and then at each step it retrieves relevant experiences from the experience bank based on the current state. Experiences that pass the similarity threshold are used to steer the hidden states.

To evaluate the effectiveness of RLC, the author tests Qwen2.5-3B/7B/14B models on 5 math benchmarks. The results show that RLC constantly outperforms the native model setting; while RLC also surpasses the baselines in most settings, these comparisons appear to be not entirely fair due to the exclusive use of Qwen3-Max model for assistance in RLC.

Critical strengths
  1. Novel method design. The Reusable Latent Correction (RLC) proposed by the authors, particularly the online intervention approach, is original as far as I know. The ablation experiments further establish the effectiveness of the two-part online intervention design. These reveal that proactively addresses the uncertainty by absorbing the prior experiences can be done at the hidden state level to improve the final performance.
  2. Comprehensive experimentations. The authors provide a wide range of experiments to facilitate the method evaluation and cross analysis. Beyond the main results on math benchmarks, they also include multiple further experiments, including but not limited to ablations of each intervention stage, an injection-layer sensitivity analysis, and the bidirectional cross-model experience bank transfer experiments. These experiments support the paper's method design choice, claims about the origin of performance gain, and help to understand its effective scope.
Critical weaknesses
  1. Potentially unfair baseline comparison through exclusive use of Qwen3-Max. RLC builds the experience bank using corrective signal from Qwen3-Max, whereas the two online-collaboration baselines are run with Qwen2.5 models as the stronger assistant. A more rigorous comparsion should use Qwen3-Max as the assistant model for the STEER and GlimpRouter baselines.
  2. Model-specific experience bank and intervention can be hard to generalize. Unlike traditional SFT where one can prepare a dataset and train different base models, the proposed RLC is model-specific as each model needs to build its own experience bank. While the authors demonstrate the some level of transferrability across two Qwen models, the generalizability of the constructed experience bank on other model family and task domains remain unknown. This limits the significance of the proposed method.
Questions for authors
  1. Could the authors report STEER and GlimpRouter with Qwen3-Max as the assisting model?
  2. Could the author report the RLC performance on any newer models such as Qwen3.5?
  3. Could the author explain why they don’t use the uncertainty metric used in the offline stage as the online intervention trigger signal?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jRKe(1/3)

Thank you for your careful review and constructive suggestions.Your comments have helped us clarify the role of the assisting LLM,strengthen the baseline comparisons,and explain the cost of adapting RLC to individual models.We address your concerns and present additional results below.

W1.Potentially unfair baseline comparison through exclusive use of Qwen3-Max.&Q1.What are the results of STEER and GlimpRouter when Qwen3-Max is used as the auxiliary model?

We apologize for the confusion caused by our description.Below,we first clarify the comparison in the original paper between RLC and directly obtaining feedback from Qwen3-Max during online inference.We then present additional results for STEER and GlimpRouter using Qwen3-Max as the assisting model to ensure a fair comparison.

(1)To clarify,Qwen3-Max is used only during offline Bank construction in RLC.RLC consists of two stages:offline construction and online inference.During offline construction,Qwen3-Max is only served as a teacher to guide the SLM.In online inference,RLC uses the SLM,consistent with the baselines.

In the original paper,we also conducted a controlled experiment in Section 5.4 to answer RQ3:Can reusable latent corrections achieve effects comparable to obtaining guidance directly from the LLM at inference time?

We replace the assisting model with Qwen3-Max,to form a direct-feedback variant under the same problem-level Router and retrieval-triggering rules.

For candidate steps that trigger intervention,the direct-feedback variant calls Qwen3-Max online to obtain guidance,whereas RLC retrieves and applies latent corrections constructed offline.Figure 4(b)summarizes the average accuracy and total number of output tokens for each model,and Table 7 in Appendix C.4 provides the results for individual datasets.Overall,in this comparison,RLC achieves higher average accuracy on 3B and 7B and comparable average accuracy on 14B with substantially fewer total output tokens.

(2)We also follow your suggestion and acknowledge that differences in the capabilities of the LLMs used in STEER and GlimpRouter can affect the fairness of accuracy comparisons.We therefore reran both baselines using Qwen3-Max as the online LLM,named as STEER(Qwen3-Max)and GlimpRouter(Qwen3-Max),and we use Qwen2.5-14B as the SLM.The results are shown below.

Table R1.Performance of STEER and GlimpRouter using Qwen3-Max as the auxiliary LLM during the online stage. Each cell reports accuracy(%)/ average output tokens.

ModelMethodMATH500GSM8KGPQAAMC24AIMEAverage
14BRLC83.00 / 583.8795.68 / 280.4552.02 / 467.1237.78 / 864.5820.00 / 1120.0757.70 / 663.22
STEER(Qwen3-Max)81.00 / 561.2495.68 / 209.8748.48 / 549.0640.00 / 874.4720.00 / 1378.3357.03 / 714.59
GlimpRouter(Qwen3-Max)80.60 / 609.8494.84 / 241.9349.49 / 553.8033.33 / 957.2716.67 / 1442.7754.99 / 761.12

As shown in Table R1,even after replacing the online auxiliary LLMs in both baselines with Qwen3-Max,RLC still achieves higher accuracy than both baselines while running only the local SLM,without calling any LLM or API.

AuthorsOpenReview ↗

Response to Reviewer jRKe(2/3)

W2. Model-specific experience bank and intervention can be hard to generalize.

We acknowledge that our method is model-specific and that transfer requires matching vector dimensions.Section 5.3(RQ2),Figure 4(a),and Table 5 in Appendix C.2 verify that the Bank has a certain degree of cross-model generalization ability.As you noted,a Bank constructed on one model cannot be used for experiments on all models.SFT uses the same dataset across models,whereas our Bank must be constructed separately for each model.However,we believe that our method can be used at a low cost,which is an advantage over SFT.Specifically,the table below compares our Bank-based approach with SFT.

MethodDataOffline preparationOnline applicationTotal cost
SFTShared datasetFine-tuning Model 1Frozen Model 1High
Fine-tuning Model 2Frozen Model 2
Fine-tuning Model 3Frozen Model 3
⋮\vdots⋮\vdots
Fine-tuning Model nFrozen Model n
RLC(Ours)Shared datasetConstruct Bank 1Frozen Model 1 + Bank 1Low
Construct Bank 2Frozen Model 2 + Bank 2
Construct Bank 3Frozen Model 3 + Bank 3
⋮\vdots⋮\vdots
Construct Bank nFrozen Model n + Bank n

Specifically,although SFT uses a shared dataset that can be applied to multiple models,it still requires separate training for each model,involving relatively expensive GPU resources and substantial costs.Our approach also requires constructing a separate Bank for each model,but this process is training-free and requires only forward passes.Once constructed,the Bank can be used directly during online inference,where model parameters remain frozen and only relevant correction vectors are retrieved in real time.Therefore,for each individual model,constructing a Bank costs less than training the model with SFT.This cost advantage allows our method to be efficiently"extended"to additional models.

Below,we compare the costs of SFT and RLC based on our experimental statistics,using 2,400 problems selected from MATH and GSM8K.We use Qwen2.5-7B-Instruct as an example.The results are shown in the following table.

MethodGPUshour/GPUGPU cost($)API cost($)Total cost($)
LoRA SFT(5 epochs)4 ×\times A800 80 GB5.518.39-18.39
RLC6 ×\times RTX 409035.077.212.27

As shown,for the 7B model,the total cost of RLC is $6.12 lower than the training cost of SFT.

AuthorsOpenReview ↗

Response to Reviewer jRKe(3/3)

Q2.Could the author report the RLC performance on any newer models such as Qwen3.5?

Due to time and computational resource constraints,we evaluated Qwen3.5-9B on 3 datasets.We also conducted experiments on Llama3.2-3B-Instruct and Gemma3-12B-Instruct from two other model providers.These experiments cover models from different providers and with different parameter counts to thoroughly verify the effectiveness and model-agnostic nature of RLC.The results are shown below:

Each cell reports accuracy(%)/ average output tokens.The Average column assigns equal weight to the 3 datasets.

ModelMethodMATH500AMC24GPQA-DiamondAverage
Llama3.2-3BNative45.20 / 715.866.67 / 1330.0222.22 / 1056.1324.70 / 1034.00
Llama3.2-3BRLC52.80 / 799.6417.78 / 1412.9633.33 / 1083.4634.64 / 1098.68
Qwen3.5-9BNative93.20 / 1321.0868.89 / 2349.4256.57 / 2588.4572.88 / 2086.32
Qwen3.5-9BRLC95.40 / 1260.0275.56 / 2479.8463.13 / 2554.5878.03 / 2098.15
Gemma3-12BNative80.80 / 833.0848.89 / 1544.4236.87 / 878.7155.52 / 1085.40
Gemma3-12BRLC85.40 / 758.4657.78 / 1475.3647.98 / 909.4763.72 / 1047.76

Compared with Native,RLC improves the average accuracy across the 3 datasets by 9.94,5.14,and 8.20 percentage points on Llama3.2-3B,Qwen3.5-9B,and Gemma3-12B,respectively.This clearly demonstrates the effectiveness and model-agnostic nature of RLC.

Q3.Could the author explain why they don't use the uncertainty metric used in the offline stage as the online intervention trigger signal?

During Bank construction,our method follows a two-stage procedure.First,JSD serves as a coarse screening signal to identify potentially problematic steps generated by the SLM.The LLM then performs fine-grained verification to determine whether correction is actually needed,allowing the step to pass if no problem is found.This design prioritizes high recall during offline construction to avoid missing potential correction signals.

Using JSD online would require repeated judgments by the LLM,resulting in unnecessary additional LLM calls.Therefore,JSD is not needed online.Instead,we directly use the similarity threshold as the trigger for online intervention,ensuring that the triggering step prefix matches a step prefix that was previously problematic,corrected by the LLM,and stored in the Bank.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer jRKe

Thank you for your response and for taking the time to reconsider our work. Regarding your concern that "the current results addressing W1 are neither complete nor sufficiently compelling," we would like to provide further clarification from two perspectives: completeness and convincingness.

1. Regarding completeness. In addition to the existing experiments with the 14B model, we have further conducted experiments using smaller models, including Llama3.2-3B,Qwen3.5-9B and Gemma3-12B. The additional experimental results are presented in the table below. As can be observed, RLC consistently achieves the best performance across these models, further demonstrating its effectiveness across different model scales.

Each cell reports accuracy (%) / mean output tokens. The five datasets receive equal weight.

ModelMethodMATH500GSM8KGPQAAMC24AIME24Average
Llama3.2-3BNative45.20 / 715.8676.95 / 220.3622.22 / 1056.136.67 / 1330.020.00 / 1498.4730.21 / 964.17
RLC(Ours)52.80 / 799.6485.97 / 214.6833.33 / 1083.4617.78 / 1412.9610.00 / 1589.2439.98 / 1020.00
STEER(Qwen3-Max)49.60 / 772.4884.00 / 229.4730.30 / 1138.2720.00 / 1379.646.67 / 1517.8338.11 / 1007.54
GlimpRouter(Qwen3-Max)50.00 / 823.1682.87 / 238.9131.31 / 1116.5215.56 / 1482.3710.00 / 1638.4537.95 / 1059.88
Qwen3.5-9BNative93.20 / 1321.0895.30 / 498.3656.57 / 2588.4568.89 / 2349.4260.00 / 3968.5374.79 / 2145.17
RLC(Ours)95.40 / 1260.0296.13 / 482.9163.13 / 2554.5875.56 / 2479.8470.00 / 3841.2780.04 / 2123.72
STEER(Qwen3-Max)94.60 / 1274.6395.83 / 508.4260.61 / 2691.3873.33 / 2534.7663.33 / 4012.8577.54 / 2204.41
GlimpRouter(Qwen3-Max)94.20 / 1356.8295.53 / 492.1762.63 / 2639.7471.11 / 2421.6366.67 / 3948.4678.03 / 2171.76
Gemma3-12BNative80.80 / 833.0894.39 / 326.5436.87 / 878.7148.89 / 1544.4223.33 / 2036.1856.86 / 1123.79
RLC(Ours)85.40 / 758.4695.00 / 311.8747.98 / 909.4757.78 / 1475.3633.33 / 1948.6263.90 / 1080.76
STEER(Qwen3-Max)83.60 / 795.3194.69 / 319.4644.44 / 946.8257.78 / 1538.2726.67 / 2084.9361.44 / 1136.96
GlimpRouter(Qwen3-Max)84.20 / 819.7494.84 / 337.1845.45 / 971.4653.33 / 1506.9223.33 / 2017.3660.23 / 1130.53
Qwen2.5-14BNative77.00 / 602.4894.47 / 290.7442.93 / 421.8731.11 / 766.8213.33 / 1122.8751.77 / 640.96
RLC(Ours)83.00 / 583.8795.68 / 280.4552.02 / 467.1237.78 / 864.5820.00 / 1120.0757.70 / 663.22
STEER(Qwen3-Max)81.00 / 561.2495.68 / 209.8748.48 / 549.0640.00 / 874.4720.00 / 1378.3357.03 / 714.59
GlimpRouter(Qwen3-Max)80.60 / 609.8494.84 / 241.9349.49 / 553.8033.33 / 957.2716.67 / 1442.7754.99 / 761.12

2. Regarding convincingness. We would like to emphasize a fundamental distinction between RLC and the compared methods. During inference, RLC does not invoke Qwen3-Max and relies exclusively on the SLM to perform reasoning. In contrast, the adapted versions of STEER and GlimpRouter require collaboration between the SLM and Qwen3-Max during inference. Despite operating solely with the SLM, RLC still outperforms both approaches. This is particularly noteworthy because RLC achieves superior performance without relying on a stronger model at inference time, providing compelling evidence of its effectiveness.

We sincerely appreciate your constructive feedback, which has motivated us to further strengthen our experimental evaluation. We hope these additional experiments and clarifications can help address your remaining concerns, and we would be grateful if you could reconsider your assessment in light of this new evidence.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

As the discussion period ends today, we would like to sincerely thank you for your time, constructive feedback, and valuable contributions to ICLR.

Following your suggestions, we have provided further clarification regarding the completeness and convincingness of our response to W1. We hope these clarifications address your remaining concerns. If any questions remain, we would be happy to provide further explanations.

If you find that our responses have satisfactorily resolved your concerns, we would be grateful if you could consider updating your assessment accordingly.

Thank you again for your thoughtful review and consideration.

Best regards,
The Authors

Thank you for your promptly reply. I appreciate your effort during this rebuttal period. I will go through your paper and the additional results carefully and decide whether to raise my score in the Reviewer-AC period.

Thank you for your consideration and your time and effort in reviewing our work. We sincerely appreciate it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.