acceptodds
Under review as a conference paper at ICLR 2027

Beyond Uncertainty: Turning Model Failures into Reusable Logic Templates for Adaptive Reasoning

Abstract

Exemplar-based chain-of-thought (CoT) prompting can improve multi-step reasoning in large language models (LLMs), but its effectiveness depends on the quality and relevance of the demonstrations for each query. Existing approaches face three limitations: demonstrations selection fails to identify genuinely difficult problems, generated reasoning paths remain noisy or redundant, and inference-time retrieval often relies on surface-level similarity rather than the underlying solution structure. To address these limitations, we propose Reusable Logic Template Activation (ReLTA), a framework for constructing and dynamically selecting reasoning demonstrations. ReLTA combines prediction errors with answer-space entropy to identify difficult examples, reconstructs and simplifies their reasoning traces into concise logic templates, and retrieves structurally relevant templates for each test query. Experiments on eight reasoning benchmarks using two reasoning backbones show that ReLTA achieves the highest accuracy on seven of eight datasets for each backbone, with an average improvement of 2.42% over Active-Prompt under the DeepSeek-Coder-V2-Lite-Base backbone. ReLTA provides a modular framework for constructing and selecting reasoning demonstrations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Chain-of-Thought Prompting, In-Context Learning, Exemplar Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5CdBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new prompting based method called ReLTA, a framework that improves LLM reasoning by improving few-shot Chain-of-Thought (CoT) examples. So the method first identifies some difficult questions using a combination of prediction error and entropy, the authors call it the difficulty score. The method then uses a larger model to generate a reasoning trace for per selected questions. At inference time, cosine similarity is used to reterive relevant examples using a reasoning-aware retriever and are then provided to the smaller model as few-shot examples. The paper shows their method across eight reasoning benchmarks and report improvements over existing prompting baselines.

Critical strengths

I think the paper is generally well written and easy to follow.

Critical weaknesses

I find several issues with the motivation of the paper.

  • The main baseline of ReLTA is active prompting, that came out in 2023/24, therefore i think ReLTA is significantly late and is not a method that I believe relevant for current age of language models.
  • On top of that the main contribution of the paper is very small, like it is mostly confined to introducing the difficulty score which is a weighted sum of already used Entropy and Error rate (the main addition of the paper). I overall think that this is very minimal contribution.
  • On top of that similar to active prompting, a larger model is required for generating CoTs. Having a method that would have not required this larger model would have improved the contribution of the paper.
  • The improvements over a simple baseline are also very minimal like 2%.
Questions for authors
  • Why are all coder models using in the evaluation when the benchmarks are not coding-based.
  • What is the reason behind using 0.7 and 0.3 for calculating difficulty score?
Reviewer_21QdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes ReLTA, a framework for building and selecting CoT demonstrations in three stages:

  1. Candidate questions are scored by a hybrid of empirical error rate and normalized answer entropy, and the top-K are selected as seeds.
  2. A stronger offline model (Qwen3-235B) solves each seed. If it fails, it derives a rationale from the reference answer. The resulting trace is simplified into a concise "logic template," accepted only if the final answer matches the reference.
  3. At test time, a reasoning-aware retriever (RaDeR) selects templates above a similarity threshold, followed by self-consistency decoding.

On eight reasoning benchmarks with Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite-Base, ReLTA outperforms reproduced baselines, including Active-Prompt and Auto-CoT, on 7 of 8 datasets per backbone.

Critical strengths
  1. Well-motivated criterion. The hybrid selection criterion is well motivated: it targets low-entropy persistent errors that entropy-only selection misses.
  2. Clear presentation. The paper is clearly written and the method is precisely specified. The authors are also candid about limitations.
  3. Controlled comparison. All baselines are reproduced under the same backbone and sampling budget, and the gains are fairly consistent across two backbones.
Critical weaknesses
  1. Teacher-model confound. The rationales come from a 235B teacher, but no control applies the same teacher pipeline to non-failure-selected questions. As a result, the gains cannot be attributed to failure-driven selection or retrieval.
  2. Mismatched baselines. All baselines use fixed exemplars, and no retrieval-based baseline is compared (e.g., kNN, EPR, UDR). ReLTA also assumes a fully labeled candidate pool, unlike Active-Prompt.
  3. Unexplained Simplification ablation. Without Simplification, accuracy falls below Zero-Shot-CoT on AQuA. On Last Letters, it drops 9.4 points even though Simplification shortens traces by only ~4.5 words. This suggests a formatting artifact rather than the benefit of removing detours.
Questions for authors
  1. Can you apply the same teacher pipeline to non-failure-selected questions (e.g., random or Active-Prompt-selected), with and without retrieval?
  2. How does ReLTA compare with retrieval-based baselines such as kNN or EPR over the full pool?
  3. Can you explain the Simplification ablation on AQuA and Last Letters, ideally with a format-controlled variant?
  4. Can you report standard deviations, and explain how θ and w1/w2 were chosen?
  5. What fraction of templates come from Reconstruction, and how often does inference fall back to zero-shot CoT?
Reviewer_dqc5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reusable Logic Template Activation (ReLTA), a framework for constructing and retrieving chain-of-thought demonstrations. It ranks answer-labeled candidate problems using a weighted combination of empirical prediction error and answer-space entropy. An offline Qwen3-235B-A22B-Instruct refinement model generates solutions, reconstructs unsuccessful ones with access to the source answer, and simplifies retained traces into concise, instance-grounded templates. At inference, a fixed RaDeR retriever selects relevant source problems and their templates; ten sampled answers are aggregated by voting. Experiments cover eight benchmarks and two reasoning backbones. ReLTA has the highest reported accuracy on seven of eight benchmarks for each backbone, with a 2.42-percentage-point average gain over Active-Prompt for DeepSeek-Coder-V2-Lite-Base. Ablations examine hybrid scoring, simplification, inference selection, and retrieval interfaces.

Critical strengths
  1. The selection criterion addresses a concrete limitation of uncertainty-based exemplar selection. A consistently wrong prediction can have low entropy, so using observed error in an answer-labeled pool is well motivated. Table 2 shows that both error-only and entropy-only selection underperform their combination while retaining the rest of the pipeline. This is direct evidence for the usefulness of the dual signal.

  2. The evaluation demonstrates useful gains across several tasks and two backbones. The main baselines are reproduced with the same final-inference model, answer parsing, and sampling budget within each backbone block. Improvements occur across arithmetic, commonsense, and symbolic tasks. The simplification ablation is particularly informative, and the paper acknowledges that its benefit may reflect both cleaner reasoning and reduced context consumption. The documented separation of candidate construction from final-evaluation files is also a positive feature.

Critical weaknesses
  1. The value of failure-driven seed selection remains incompletely isolated. ReLTA combines a large offline teacher with query-dependent retrieval, whereas the principal prompting baselines use their respective exemplar sets. The within-pipeline scoring ablations support combining error and entropy, but a randomly selected seed pool processed with the same teacher and retrieval pipeline is missing. This control would clarify how much additional benefit comes from targeting model failures rather than constructing and retrieving high-quality demonstrations more generally.

  2. The precision of the reported gains and the exact prompting protocol are difficult to assess. Table 1 averages three runs without reporting their variability or explaining which components are resampled, and the ablation tables provide no uncertainty estimates. Several gains are small enough that this matters. The appendix supplies construction pseudocode, but does not give the actual Generation, Reconstruction, and Simplification prompts. The exact RaDeR checkpoint and how the shared threshold and hybrid weights were chosen are also unspecified. Table 3 compares native retrieval protocols with different filtering behavior, which the paper acknowledges; it therefore supports that implementation's downstream utility more directly than a controlled conclusion about structural retrieval. These omissions limit both reproduction and interpretation of the claimed contribution.

  3. Limited demonstrated portability and substantial construction costs weaken the practical case for this training-free method. ReLTA requires an answer-labeled candidate pool, up to 10,000 target-model generations for seed scoring, and Qwen3-235B-A22B-Instruct calls to construct templates. Pools are rebuilt for each backbone, and adaptation to a new domain may require repeating this pipeline. Reusing GSM8K templates on three arithmetic benchmarks provides some transfer evidence, but reuse across backbones and substantially different domains is not established. Given average gains of approximately 1.76 and 2.42 percentage points over Active-Prompt, the additional preparation cost may outweigh the benefit in low-volume deployments. The paper should report comparative end-to-end costs, demonstrate reuse without rebuilding the pool, and quantify how many queries are needed to amortize construction. Avoiding parameter updates alone does not establish inexpensive, readily transferable adaptation.

Questions for authors
  1. How does ReLTA compare with a randomly selected seed pool passed through the same teacher, answer checks, and RaDeR retrieval? Please keep pool size and exemplar budget comparable, or explicitly account for differences, to isolate the benefit of failure-driven selection.

  2. Please provide per-run results, state whether each run rebuilds the pools or only resamples final answers, and release the exact refinement prompts and retriever identifier. Were the threshold and hybrid weights selected using data separate from final evaluation? An account of actual exemplar counts and prompt truncation would also help interpret the simplification and retrieval ablations.

  3. Please clarify whether exact- and near-duplicate checks were performed between candidate and evaluation files, particularly in the cross-dataset setting.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period is coming to an end, I kindly encourage you to review and respond to the authors' rebuttal as soon as possible.

Thank you very much for your time and valuable contributions to the review process.

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.