- Rating
- 1
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Jailbreak for Uncertainty (J4U), which applies jailbreak-inspired prompt transformations to benign questions and estimates confidence from the frequency of the majority answer across multiple samples. The transformations include random suffixes, ASCII-art substitution, and text decomposition and reconstruction. The paper interprets these transformations as relaxing RL-induced constraints, motivated by a KL-regularized policy model. Experiments on four models and three main datasets report improvements in calibration metrics over repeated sampling and two prompting baselines. Additional analyses examine answer entropy, reasoning-trace similarity, sampling budget, and performance on GSM8K.
Critical strengths
The method has a simple deployment interface (a prompt-engineering method). It requires only textual queries and responses, without model modification or access to logits.
The empirical calibration results merit investigation. The paper reports improvements beyond ECE, including Brier score and NLL, across several model–dataset combinations.
Critical weaknesses
I am confused when reading the paper perhapse I didn't understand the full picture.
The abstract moves from
"Focusing on question-answering for LRMs, we show that existing black-box methods, such as paraphrase-based self-consistency and confidence verbalization, offer little to no improvement over simple repeated sampling"
to
", suggesting that alignment suppresses useful output variability."
I find it hard to understand the logic here. How was this conclusion reached? The paper didn't perform SFT or RL, nor did the authors compare a base model against a chat model. The actual comparison was against various perturbation methods, yet the conclusion shifted to alignment affecting model capabilities. I’m not really sold on this explanation. The paper mentions "RL" 95 times, yet the actual experiment doesnt seems to be related to RL. How do the authors prove that "jailbreak-style" prompts don't introduce confounding effects related to RL alignment? Can the prompts that authors used actually trigger a jailbreak in the first place? I don't see any experiments showing whether the base models exhibited the same response to the perturbations they tested. Sec 3 defines Pi_ref as the base model, and Pi_RL as the RL model but all the experiments are done only on Pi_RL, Pi_ref only exists in the writing.
The experiments are done on small language models (the largest one here may be luna). The method is basically a string level method (requires no adapter or training), and I believe its easy to scale.
The writing is poor. Sec 3 is quite long but the content seems to be basic definitions (e.g. ECE) . Also, the paper lacks a diagram illustrating the methodology; Figures in the appendix are with tiny font sizes
Questions for authors
The main issue is that I don't see what purpose J4U serves beyond the explanation provided in the paper. Looking at the metrics, the accuracy remains roughly the same under the prompt-based setup, though the ECE drops. If the main conclusion is that RL reduces output diversity, then the experiments and the conclusion don't align.
5 comments
(cont)
6. The reference policy is not evaluated, and base models might respond similarly. We have not compared the released models with their corresponding pre-alignment reference policies, so we cannot determine whether the base models would respond similarly. Section 5.4 now explicitly states that answer entropy and trace similarity characterize J4U’s effects but do not establish movement toward .
7. The experiments use small language models. The evaluation includes three open-weight models from 4B to 32B parameters and GPT-5.6 Luna, accessed through a production API. The evidence therefore extends beyond the small open-weight models. On GPT-5.6 Luna and SynthPAI, ART reduces ECE from 0.32 to 0.26 and NLL from 9.86 to 7.45, while accuracy remains 0.61 (Table 7). J4U requires no model modification in either setting and is applicable to any model. These results demonstrate applicability across the evaluated models, although they do not isolate the effect of model size.
8. Section 3 is long, the method lacks a diagram, and appendix figures are difficult to read. We shortened Section 3 to approximately one page by defining the theoretical mode, accuracy indicator, and ECE inline and removing the separate definition of perfect calibration. Figure 1 now illustrates the complete procedure: prompt transformation, repeated sampling, majority prediction, and confidence estimation. We also enlarged the axis labels, tick labels, and legends in Figures 2–4. The new introduction paragraph explains why confidence calibration matters in diagnostic support and text anonymization.
Questions for Authors:
1. What purpose does J4U serve when accuracy stays roughly the same but ECE decreases? The objective of J4U is to improve calibration without degrading accuracy. Improving accuracy is not required for an uncertainty estimator to be useful. Accuracy measures how often the model is correct; calibration measures whether its confidence agrees with its probability of correctness. Similar accuracy can therefore coexist with substantially more reliable confidence scores.
For example, on Qwen3-4B and SynthPAI, RS achieves 56% accuracy and ECE 0.34. With the same 10 samples per question, J4U-PROG achieves 58% accuracy and ECE 0.23; Brier score and NLL also improve (Table 4). A user needs to know both what answer the model gives and how much confidence to place in it. Similar accuracy does not make these confidence estimates equally useful.
2. If the main conclusion is that RL reduces output diversity, the experiments and conclusion do not align. The main empirical conclusion is that J4U improves black-box confidence estimation on the evaluated LRMs. Across the twelve model–dataset pairs, PROG and ART reduce ECE by 15.8% and 13.8% on average relative to RS, compared with 3.1% for REPHRASE (Table 1). The revised abstract makes this practical contribution explicit: better-calibrated confidence estimates obtained through text queries and responses alone.
Despite the lengthy response, posted on the final day of rebuttal, my main concern remains unresolved. The paper’s theoretical explanation centers on RL alignment, and I had expected experiments directly testing this explanation. The authors now explicitly acknowledge that they have not established that J4U satisfies their definition of a relaxation operator. Presenting this connection as a hypothesis leaves the link between the theory and the actual method unsupported. Similarly, citing jailbreak attacks from 2024 does not demonstrate that these adapted prompts successfully jailbreak the models evaluated in late 2026, given how quickly attacks and defenses evolve. Calling them “jailbreak-inspired” explains their origin, but supplies no experimental evidence that they bypass alignment on these models.
In light of these, I will maintain my rating and increase my confidence
Thank you for your detailed review and for recognizing the practical interface and empirical calibration results. J4U addresses a limitation of repeated-sampling confidence: a model can repeatedly give the same incorrect answer, so strong agreement need not imply correctness. J4U applies jailbreak-inspired transformations to benign questions and uses the frequency of the majority answer as its confidence score. The experiments test whether this score better reflects the probability that an answer is correct while maintaining comparable accuracy. A change in the majority answer is not required for this improvement.
We revised the abstract and Section 5.4 to clarify this distinction, added practical motivation and Figure 1 to illustrate the mehtodology and its objectives, and expanded the comparisons with REPHRASE, temperature-adjusted repeated sampling, and a logit-based baseline. All additions and changes are visible in blue in the revised paper. Please find below our responses to your concerns.
Critical Weaknesses:
1. The abstract suggests that alignment suppresses useful variability. We revised the abstract to distinguish the hypothesis motivating J4U from the experimental findings. RL-induced sharpening is motivated by prior work and a hypothesis that drives the design of J4U; our empirical finding is that J4U improves confidence calibration on the evaluated LRMs. The scope of the theoretical explanation is detailed in point 3.
2. What comparisons with perturbation methods establish. Comparing against existing UQ methods, including prompt perturbations, directly tests J4U’s practical contribution. These comparisons establish the relative effectiveness of the UQ methods; they do not identify the effect of alignment. We strengthened this evaluation with a paired comparison against REPHRASE (Appendix D.5, Table 15). Averaged over the three datasets, J4U-ART and J4U-PROG improve ECE, Brier score, and NLL relative to REPHRASE on each of the four models. The improvements are statistically significant in all 12 model–metric comparisons for ART and 10 of 12 for PROG.
We also added a simple logit-based baseline on Qwen3-4B and gpt-oss-20b (Appendix D.4). J4U achieves lower ECE and Brier score on every evaluated model–dataset pair, without requiring access to token probabilities. These comparisons strengthen the empirical contribution independently of the proposed explanation involving alignment.
3. RL appears throughout the paper, but the experiments do not involve RL training. RL defines the policy model analyzed in Section 4. Under its assumptions, Proposition 4.2 establishes movement toward the reference policy, and Theorem 4.3 shows how calibration can improve while the most probable answer remains unchanged: a moderate relaxation reduces excessive confidence, with strict calibration improvement guaranteed in the large-sample limit. This result motivates the search for prompt transformations with this behavior. These guarantees apply to operators satisfying Definition 4.1. We have not established that J4U satisfies this definition. The experiments directly evaluate its calibration performance at a finite sampling budget on released models (i.e. πRL). They support the practical benefit of J4U, while identifying relaxation as its mechanism requires further evidence. We clarified this distinction in the abstract and Section 5.4.
4. Could the improvements arise from effects unrelated to alignment? The new experiments test a concrete alternative explanation: whether simply increasing answer diversity reproduces J4U’s performance. We evaluated temperature-adjusted RS on Qwen3-4B and two datasets, with 100 inputs per dataset and ten samples per input (Appendices D.6–D.7).
On SuperGPQA, J4U-PROG and RS at T=2.7 have similar mean answer entropies, 0.85 and 0.82. PROG achieves lower ECE, 0.22 versus 0.29, and higher accuracy, 0.46 versus 0.35. For ART, the closest RS run matches its ECE of 0.29.
On SynthPAI, the closest RS run reaches entropy 0.35 versus 0.36 for ART and PROG. Its ECE is 0.30, compared with 0.23 and 0.24, respectively. Only 34.3% of its outputs contain a valid label, versus 100% for J4U. Generation limits were 16,384 tokens for RS and 4,096 for J4U.
Thus, similar mean answer entropy does not necessarily produce similar calibration or accuracy. Increasing temperature does not consistently reproduce the performance of J4U-ART and J4U-PROG in these experiments.
5. Can these prompts actually trigger a jailbreak? ART and PROG are adapted from published jailbreak attacks (Jiang et al., 2024; Kang et al., 2024), while SUFFIX uses a simplified random suffix. “Jailbreak-inspired” describes the origin of these transformations. Here, the transformations are applied to benign questions and evaluated for confidence estimation. Their usefulness for UQ is measured directly by the calibration results, rather than inferred from attack success.