acceptodds
Under review as a conference paper at ICLR 2027

Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?

Abstract

Agent harnesses often improve language models with human-designed workflows, but as models grow more capable, unreliable guidance can increasingly constrain their execution. We call the ability to benefit from useful guidance while overriding unreliable guidance **thinking outside the box**. We introduce Box-Bench, which holds the model and task fixed while varying workflow reliability to isolate how models regulate their reliance on guidance. On Box-Bench, frontier models often benefit from reliable guidance but remain vulnerable when it is misleading or becomes unreliable. To test whether this capability can be learned, we train two open-weight models using bad workflows, reserving good workflows for evaluation. We explore two complementary training strategies: counterfactual supervised fine-tuning improves robustness, while outcome-based reinforcement learning can shift the balance toward greater use of helpful workflows. We further find that this behavior extends beyond workflows to other forms of external information, improving peer correction and robustness to corrupted memory. Together, our results identify selective reliance on fallible external information as a dimension of agent reliability not captured by task performance alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent harness, Selective workflow reliance, Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9DMDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether language models can use helpful external workflows while overriding misleading guidance. It introduces Box²-Bench, which holds the model, task, environment, and evaluator fixed across five conditions: no, good, bad, partial, and mixed workflows. Matched comparisons separate useful-guidance utilization, robustness to bad guidance, and recovery after guidance stops or becomes misleading. Experiments show that good workflows often help while bad workflows hurt. Training two open-weight models on bad-workflow inputs improves robustness, but SFT can also suppress useful-guidance utilization; outcome-based RL partly restores it. Peer-information and memory experiments provide preliminary evidence of transfer beyond workflows.

Critical strengths

The strongest aspect is the controlled evaluation design. Varying workflow quality while keeping the task and model fixed helps distinguish ordinary task failure from failure caused by misleading guidance. The Partial and Mixed conditions are especially useful because they share a helpful prefix and differ in whether guidance stops or continues incorrectly. Separating utilization from robustness also captures a practical distinction that raw accuracy misses: a model can benefit substantially from a good workflow and still be highly vulnerable to a bad one.

The training analysis identifies a meaningful failure mode of bad-workflow training. A model can improve by selectively overriding incorrect steps or by ignoring workflows altogether, and the held-out good-workflow condition distinguishes these behaviors. The AIME result, where SFT improves robustness but makes useful-guidance utilization negative before RL restores positive utilization, illustrates the trade-off clearly. The transfer probes also examine behavior beyond final accuracy; in particular, archive verification under corrupted memory offers a useful indication of how the trained model handles fallible context.

Critical weaknesses

The benchmark uses a specific workflow-error distribution produced by a fixed external generator and filtered by verification. This gives experimental control, but it leaves open whether good and bad workflows differ in surface style or structure in ways that models can exploit. Such cues could partly explain apparent selective reliance without requiring an assessment of guidance reliability. Evaluations with other generators, human-written workflows, or style-matched errors would help resolve this concern. The verified conditions also represent cleaner reliability interventions than the ambiguous, stale, or partially correct guidance encountered in deployment, so the conclusions should remain scoped to the controlled setting.

The statistical and training evidence is limited. Some evaluations use 30-problem subsets, making point estimates sensitive to a few examples, yet uncertainty reporting is sparse. Learning is demonstrated with Qwen3-4B on AIME 2026 and Qwen3.5-9B on WebShop, which supports a proof of concept more readily than a general training recipe. Cross-context transfer is also mixed: multi-agent episode success rises from 18.00% to 22.67% with SFT but reaches only 18.67% with SFT+RL, and the memory probe uses a 30-question LongMemEval-V2-Small subset with a modified archive protocol. These are informative probes, but broader claims need replication across models, tasks, and training seeds.

Questions for authors

Q1. How sensitive are the benchmark effects to the workflow generator and to surface differences between good and bad guidance? Could you evaluate a subset using another planner or human-written workflows, together with a style-matching control or a classifier that predicts workflow labels from surface form? Please include confidence intervals or paired uncertainty estimates for the utilization, robustness, and Partial-versus-Mixed effects, since small subsets can make these comparisons unstable.

Q2. Does the SFT-to-RL trade-off persist across model sizes, domains, and training seeds, and what explains the lower multi-agent success of SFT+RL relative to SFT alone? Trajectory-level diagnostics could help distinguish successful recalibration from renewed harmful reliance. Given the size and specialized protocols of the peer and memory experiments, could you clarify which cross-context claims are supported by the current evidence and frame the remaining results as preliminary transfer probes?

4 comments

Thank you for the encouraging feedback. We appreciate your recognition of the paper's controlled evaluation design, training analysis, and transfer studies. We address your concerns point by point.

W1 & Q1: Ablation workflow generator.

Response: We address this concern from two aspects: workflow generator robustness, controlled comparisons of workflow content.

(1) Sensitivity to workflow generators. To test whether our findings depend on the workflow generator, we conduct additional experiments using GPT-6.1-Sol to generate workflows and evaluate them on both AIME 2025 and AIME 2026. We report mean accuracy and standard deviation over eight rollouts.

Across both datasets, we observe a consistent pattern with our original results: SFT improves Bad-workflow performance while reducing Good-workflow performance, and subsequent RL further improves Bad and Mixed performance while maintaining comparable Good-workflow performance. These results suggest that our findings are not specific to the original workflow generator. The results are in Table 5 of the Appendix B.1 (Ablations of Workflow Generator) in the updated PDF.

AIME 2025 with GPT-6.1-Sol generator :

ModelNoneGoodBadPartial-4Mixed-4
Base63.33 ± 6.6795.83 ± 3.8835.00 ± 7.9782.92 ± 3.3052.92 ± 5.76
SFT58.33 ± 5.9190.42 ± 4.5250.83 ± 7.2978.75 ± 2.4859.17 ± 6.61
SFT+RL61.25 ± 6.6590.42 ± 3.7552.50 ± 6.6177.50 ± 4.2760.42 ± 9.33

AIME 2026 with GPT-6.1-Sol generator :

ModelNoneGoodBadPartial-4Mixed-4
Base62.50 ± 8.8688.33 ± 1.7823.75 ± 5.4776.67 ± 3.0938.33 ± 7.77
SFT58.33 ± 4.7184.58 ± 2.4850.00 ± 5.3575.00 ± 3.9850.42 ± 4.86
SFT+RL56.25 ± 7.0083.75 ± 6.0352.50 ± 4.6373.33 ± 5.3550.83 ± 4.96

(2) Surface-level confounds. To probe this, Box²-Bench includes Partial and Mixed conditions, which share the same helpful prefix but differ in whether misleading steps are appended. Thus, Δswitch=SM−SP\Delta_{\mathrm{switch}}=S_M-S_P measures the impact of misleading information while controlling for the helpful content, providing a finer-grained evaluation of selective workflow use.

AuthorsOpenReview ↗

Rebuttal (continued)

W2: SFT-to-RL trade-off and cross-context transfer.

Response: We address two aspects of this concern. First, we clarify the SFT-to-RL trade-off with additional empirical evidence from AIME and WebShop. Second, we provide additional behavioral diagnostics in multi-agent and memory settings to examine cross-context transfer.

Reason of SFT-to-RL trade-off:

  • (1) Theoretical evidence. SFT trains the model to produce a prescribed correct trajectory given a misleading workflow (Eq. 4). RL, in contrast, optimizes task success without prescribing a target trajectory (Eq. 5). Through task success feedback, model may learn to selectively identify and use information from the workflow. This ability also useful for Good workflows. Similarly, using workflow information may introduce a trade-off: it can help the model benefit from useful guidance while increasing its exposure to misleading steps.
  • (2) Empirical evidence. We note that SFT+RL consistently outperforms SFT across all workflow conditions on WebShop (Fig3 in paper). On AIME 2025 and AIME 2026 with the GPT-6.1-Sol generator, we also observe improvements in mean accuracy on Bad and Mixed workflows (Tab1-2 in W1). These results suggest that the model can learn to identify and use useful workflow information. However, this remains a challenging capability. Performance may vary depending on how misleading the workflow is and how reliably the model can identify relevant information and use it correctly.

Robustness of cross-context transfer:

To further assess the robustness of our cross-context transfer results, we conduct additional evaluations with five random seeds and report mean ± SD.

Multi-agent transfer (EoM). Episode success rates remain relatively stable across training stages. However, under BAD conditions, SFT+RL achieves more repairs (7.20 ± 2.68 vs. 4.80 ± 3.35) and fewer corruptions (0.80 ± 1.10 vs. 3.20 ± 1.79) than SFT. This reveals a behavioral change in how agents handle misleading peer information, even when improvements in overall episode success are not consistent. Due to rebuttal budget, we use 10-problem subsets with five seeds.

Memory transfer. We observe a clearer change in verification behavior. After SFT+RL, archive-tool usage is higher under BAD than GOOD memory (72.67% vs. 66.67%), reversing the trend in the base model (64.67% vs. 73.33%). This suggests that the model more actively verifies potentially misleading memories.

These results provide additional evidence of behavioral transfer beyond workflow tasks. Given the rebuttal compute budget, rerunning the main experiments across independent training seeds was not feasible. We will add additional experiments to further consolidate the evidence in next version.

Multi-agent transfer (EoM):

ConditionModelPass RateRepairCorruption
RAWBase18.80 ± 2.282.00 ± 1.411.60 ± 1.67
RAWSFT20.00 ± 2.456.40 ± 2.610.80 ± 1.79
RAWSFT+RL19.60 ± 2.193.20 ± 1.792.80 ± 1.79
BADBase17.60 ± 2.192.00 ± 1.412.00 ± 1.41
BADSFT19.60 ± 2.614.80 ± 3.353.20 ± 1.79
BADSFT+RL17.60 ± 2.617.20 ± 2.680.80 ± 1.10

Memory transfer:

ModelRAW AccuracyRAW Tool RateGOOD AccuracyGOOD Tool RateBAD AccuracyBAD Tool Rate
Base24.00 ± 7.96100.00 ± 0.0094.67 ± 1.8373.33 ± 6.2410.67 ± 1.4964.67 ± 9.89
SFT27.33 ± 2.79100.00 ± 0.0092.67 ± 3.6572.67 ± 7.2312.00 ± 6.0671.33 ± 6.91
SFT+RL27.33 ± 4.94100.00 ± 0.0092.00 ± 4.4766.67 ± 8.5012.00 ± 2.9872.67 ± 4.35

Thanks authors for the detailed response. They have largely address my concerns on my side. Thanks.

Thank you again for your time and thoughtful engagement with our submission! Your valuable suggestions have greatly helped us improve the quality of our work!

Reviewer_8ymYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether LLMs can use helpful workflow guidance while resisting misleading guidance. It introduces Box2-Bench, which evaluates the same tasks under good, bad, partial, and mixed-workflow conditions. The authors also train models on misleading workflows using counterfactual SFT followed by RL. Their experiments cover reasoning, coding, search, and tool use, with additional transfer evaluations involving peer information and memory. SFT improves robustness to bad workflows but reduces the benefit of good workflows; RL partially changes this tradeoff.

Critical strengths
  1. The comparison between partial and mixed is useful. Both conditions share the same helpful prefix, so the comparison measures the additional effect of misleading suffix. Freezing workflows across evaluated models also makes the comparisons easier to interpret.

  2. The training results expose a meaningful tradeoff. In Table 3, SFT substantially reduces the damage from bad workflows on both AIME and WebShop, while also reducing gains from good workflows. Reporting both sides gives useful information about what the training changes.

Critical weaknesses

Limited evidence for selective reliance and transfer.

In Table 4, SFT improves Bad-memory accuracy from 10.0% to 14.4%, but no-memory accuracy improves more, from 21.1% to 28.9%. The penalty from corrupted memory therefore grows.

Multi-agent gains are also inconsistent across training stages. With only 30 distinct problems per transfer evaluation and no uncertainty estimates, the broader generalization claim remains weak.

Questions for authors

How should we interpret the memory robustness claim given that the bad-versus-no-memory accuracy gap becomes larger after training in Table 4?

1 comment

Thank you for the encouraging feedback. We appreciate your recognition of the clear presentation, the informative comparison between Partial and Mixed conditions, and the meaningful tradeoff revealed by our training results. We will address your concerns point by point.

W1 & Q1: Interpretation of the increased Bad-memory versus No-memory performance gap after SFT.

Response: Our goal is to benchmark models’ ability to think outside the box and investigate how to train models to develop this capability. We view thinking outside the box as a meta-capability across context-rich settings. Despite being trained exclusively on WebShop workflows without memory-specific supervision (Appendix D.1), our SFT model improves accuracy under both No-memory (21.1% → 28.9%) and Bad-memory (10.0% → 14.4%) conditions. While the smaller gain under Bad memory indicates that corrupted context remains challenging, the positive improvement demonstrates that workflow training can transfer to memory-augmented tasks.

ModelGood Tool RateBad Tool Rate
Base72.2%65.6%
SFT72.2%72.2%
SFT + RL66.7%74.4%

More interestingly, we observe a behavioral transfer in tool usage. After SFT+RL, the model invokes verification tools more frequently under Bad than Good memory (74.4% vs. 66.7%), reversing the trend in the Base model (65.6% vs. 72.2%). This suggests that workflow training also encourages model to seek external evidence when facing potentially misleading information, even without memory-specific training.

W2: Limited evidence for memory robustness and cross-context transfer.

Response: We agree that more problem and multi-seed reporting would further strengthen the empirical results, but running repeated full evaluation experiments is beyond the scope of the rebuttal budget. We choose ten subset of AIME2026 and random 5 seeds to consolidate the result.

To further assess the stability of our results, we report additional mean and standard deviation across five random seeds on AIME 2026. We will extend the memory and multi-agent evaluations with larger test sets and multiple seeds in the next version.

ConditionModelPass Rate (Mean ± SD)Repair (Mean ± SD)Corruption (Mean ± SD)
RAWBase18.80 ± 2.282.00 ± 1.411.60 ± 1.67
RAWSFT20.00 ± 2.456.40 ± 2.610.80 ± 1.79
RAWSFT+RL19.60 ± 2.193.20 ± 1.792.80 ± 1.79
BADBase17.60 ± 2.192.00 ± 1.412.00 ± 1.41
BADSFT19.60 ± 2.614.80 ± 3.353.20 ± 1.79
BADSFT+RL17.60 ± 2.617.20 ± 2.680.80 ± 1.10
Reviewer_pRCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper discusses whether language models can distinguish between a helpful external guidance on a task and a non-helpful one reliably. The paper is focused mostly on workflows but it also covers if the similar behavior expands to memory and multi agent setups, where agent’s ability to recognize an unreliable input from their teammates (other agents) or an unreliable memory could be critical. The main contributions of the paper are as follows: 1) The paper provides a benchmark to identify LLM agents ability on discarding the unreliable workflows and following the reliable workflows, evaluating some open weight and closed weight models on this benchmark. 2) To make the agent robust against bad workflows, they design a supervised learning mechanism and a reinforcement learning method. Meanwhile, they also analyze whether the agents are still performing well or not. 3) The analyze whether their method for workflow extends to the memory-based reasoning and multi-agent reasoning or not.

Critical strengths

First, the paper is well written and easy to follow. Second, the paper's motivation in memory augmented reasoning and multi-agent reasoning is an interesting viewpoint for the current problem. Third, the paper in Section 2 provides analysis for different open weighted and closed wieght models.

Critical weaknesses

The first weakness of the paper is related to Section 3. The model is trained on AIME and WebShop and then evaluate on the same datasets. For verifying whether the model can generalize well to other contexts and benchmarks, it should be evaluated on other benchmarks beyond these two as well.

Second, Equation 4 uses only the bad workflows. Although the authors mentioned that they do not train on good workflows as they want to see generalization, I disagree with this point of view. One consequence of Equation 4 could be that the model completely ignores the workflow. Table 3 also confirms this by the decrease in model performance in good workflows. Thus, probably a better loss function is a contrastive learning loss. For evaluating the generalization of the model, one can use different benchmarks and datasets.

Third, authors do not provide any motivation for performing RL in Equation 5 and even doing so after SFT. Why don't we start with RL from scratch? But even after justifying these questions, the more important question is that why RL (in Equation 5) results in improvement in the performance of the model on good workflows and decreases the performance on the bad workflows according to Table 3? In fact, in equation 5, we never sample the good workflow; thus it is not clear why RL here helps with the good workflow while it degrades the performance obtained on bad workflows (while these bad workflows (according to Equation 5) is the objective of the RL here).

Fourth, AIME is a very small dataset; therefore changes in one or two questions can change the performance percentage significantly.

Fifth, the results presented in tables 1-3 could be discussed in more detail. For example, based on table 1, it seems that partial workflow (SPS_P) sometimes outperforms Good workflow (SGS_G) but section 2.3 does not elaborate on this.

Sixth is that a related work has not been discussed: reference [1] works on the same problem and designed a benchmark similar to the setup of this paper. The paper needs to clarify their contribution with respect to this reference.

[1] Fu Y, Qiu R, Wang X, Sansom J, Prabhu SA, Tang H, Kim J, Sohn S, Lee H. Beyond blind following: Evaluating robustness of llm agents under imperfect guidance. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) 2026 Mar (pp. 6591-6618).

Questions for authors

Please see the "Critical Weaknesses" section. Specifically, I am interested to know: 1. Why RL in Equation 5 is helping the performance on good (SgS_g) workflows in table 3 and degrades the performance on bad workflows (SBS_B)? 2. What is the contribution of the current paper compared to the reference [1]?

[1] Fu Y, Qiu R, Wang X, Sansom J, Prabhu SA, Tang H, Kim J, Sohn S, Lee H. Beyond blind following: Evaluating robustness of llm agents under imperfect guidance. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) 2026 Mar (pp. 6591-6618).

4 comments

Thank you for the encouraging feedback. We appreciate your recognition of the paper's clear presentation, its interesting extension to context-rich settings such as memory and multi-agent, and its evaluation covering both open- and closed-weight models. We will address your concerns point by point.

W1: Training & Evaluation Detail

Response: We would like to clarify training and evaluation are already separated in both AIME and WebShop. We show that:

  • AIME: SFT uses 2,060 source problems from InT-SFT, and RL draws from the same training pool. AIME 2026 is exclusively used for evaluation (Appendix A.1.1 and A.2).
  • WebShop: We follow the official train/test split, using training tasks (indices 1500–12086) for SFT and RL and the 500 official test tasks (indices 0–499) for evaluation (Appendix A.1.2 and A.2).

To further address the generalization concern, we additionally evaluate our models on AIME 2025 and AIME 2026 using GPT-6.1-sol generator. For each problem, we sample 8 rollouts and use majority voting to determine the final answer. The results are as follows:

For AIME 2025:

ModelNoneGoodBadPartialMixed
Base63.33 ± 6.6795.83 ± 3.8835.00 ± 7.9782.92 ± 3.3052.92 ± 5.76
SFT58.33 ± 5.9190.42 ± 4.5250.83 ± 7.2978.75 ± 2.4859.17 ± 6.61
SFT+RL61.25 ± 6.6590.42 ± 3.7552.50 ± 6.6177.50 ± 4.2760.42 ± 9.33

For AIME 2026:

ModelNoneGoodBadPartialMixed
Base62.50 ± 8.8688.33 ± 1.7823.75 ± 5.4776.67 ± 3.0938.33 ± 7.77
SFT58.33 ± 4.7184.58 ± 2.4850.00 ± 5.3575.00 ± 3.9850.42 ± 4.86
SFT+RL56.25 ± 7.0083.75 ± 6.0352.50 ± 4.6373.33 ± 5.3550.83 ± 4.96

Partial: Good-4; Mixed: 4G+4B. All results are over 30 problems.

W2: SFT objective and contrastive learning

Response: We chose Bad-only counterfactual SFT for its simplicity and training efficiency, primarily targeting robustness to misleading guidance. We acknowledge that this objective may introduce a trade-off in helpful workflow utilization, as reflected by the lower Good and Partial accuracy in our results. However, this does not necessarily mean that the model ignores workflow information. As shown in Figure 4 and Tables 7–8, performance generally improves as the amount of helpful workflow content increases, suggesting that models retain some ability to benefit from useful guidance. This limitation also motivates our subsequent RL stage, which aims to further improve selective workflow utilization.

Contrastive learning is also a reasonable alternative, but typically requires paired examples and additional training computation, which motivated our initial choice. To further explore this alternative, we have conducted an initial additional experiment using a contrastive learning objective.

ModelNoneGoodBadPartialMixed
Base66.780.046.786.773.3
Original SFT73.370.066.776.773.3
Contrastive SFT60.063.360.063.363.3

Partial: Good-4; Mixed: 4G+4B. All results are over 30 problems. Contrastive results are preliminary.

Given the limited rebuttal time, we view the results as preliminary. Contrastive SFT also improves robustness to Bad workflows. We will further investigate and consolidate into a dedicated ablation section in the revised paper.

AuthorsOpenReview ↗

Rebuttal (continued)

W3 & Q1: Motivation and behavior of SFT-to-RL

Response: We will discuss our SFT-to-RL approach from two aspects: the motivation for adopting this training pipeline and the effects of RL on workflow utilization and robustness.

Design of RL

  • (1) Our training pipeline follows common practice, with SFT providing a cold start for subsequent RL [1–5].
  • (2) We adopt SFT before RL for efficiency. As shown in Figure 3, our SFT data are simple and scalable to construct, providing direct supervision without relying on sparse task-level rewards.
  • (3) Comparing against RL from scratch requires substantial additional training compute, which is beyond our rebuttal budget. We plan to add this in next version.

RL performance

  • (1) Why can RL affect Good-workflow performance despite training only on Bad workflows? SFT trains the model to produce a prescribed correct trajectory given a misleading workflow (Eq. 4). RL, in contrast, optimizes task success without prescribing a target trajectory (Eq. 5). Through task success feedback, model may learn to selectively identify and use information from the workflow. This ability also useful for Good workflows. Similarly, using workflow information may introduce a trade-off: it can help the model benefit from useful guidance while increasing its exposure to misleading steps.
  • (2) Empirical evidence. We note that SFT+RL consistently outperforms SFT across all workflow conditions on WebShop (Fig3 in paper). On AIME 2025 and AIME 2026 with the GPT-6.1-Sol generator, we also observe improvements in mean accuracy on Bad and Mixed workflows (Tab1-2 in W1). These results suggest that the model can learn to identify and use useful workflow information. However, this remains a challenging capability. Performance may vary depending on how misleading the workflow is and how reliably the model can identify relevant information and use it correctly.

We will revise our discussion to better reflect this behavior and performance of RL.

W4: Statistical reliability of AIME results

Response: In our original evaluation, we use 8 rollouts per problem with majority voting to reduce randomness across individual rollouts. To further address the limited size of AIME 2026, we additionally evaluate on AIME 2025. The results are presented in our response to W1.

W5: Interpretation of Partial vs. Good workflow results

Response: This phenomenon is consistent with prior findings on partial harnessing [6]. Intuitively, even when the guidance is correct, additional workflow steps introduce execution risks, which may outweigh their marginal benefits and ultimately reduce task performance. We will revise Section 2.3 for better explaination in the next revision.

W6 & Q2: Insufficient discussion of distinctions from prior work.

Response: Thank you for pointing us to MIRAGE [7], which introduces a benchmark evaluating LLM robustness under imperfect guidance, including incomplete, incorrect, misaligned, and underspecified instructions. Our work builds on this setting but focuses on selective workflow utilization.

  • Box²-Bench evaluates selective workflow use. For example, Good vs. None measures the benefit of helpful guidance. Partial vs. Mixed measures the effect of adding misleading steps to the same helpful prefix.
  • We study whether selective workflow use can be learned and transferred. We train with counterfactual SFT and outcome-based RL, then test transfer to memory and multi-agent settings.

We will clarify these contributions in Related Work.

[1] Li, Xinran, et al. "Getting your LLMs ready for reinforcement learning with lightweight SFT." The Fourteenth International Conference on Learning Representations. 2026.

[2] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.

[3] Wei, Yana, et al. "Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning." Advances in Neural Information Processing Systems 38 (2026): 4463-4494.

[4] Zheng, Tong, et al. "Parallel-r1: Towards parallel thinking via reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.

[5] Feng, Kaituo, et al. "Video-r1: Reinforcing video reasoning in mllms." Advances in Neural Information Processing Systems 38 (2026): 99114-99137.

[6] Wang, Boyuan, et al. "Harnesses for Inference-Time Alignment over Execution Trajectories." arXiv preprint arXiv:2605.21516 (2026).

[7] Fu, Yao, et al. "Beyond blind following: Evaluating robustness of llm agents under imperfect guidance." Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Thank you so much to the authors for providing answers to the weaknesses. While my concerns on the rest of them are addressed, I am still not convinced about W2 and the latter part of W3. I still think that the defined objectives function has no terms for Good workflows. Hence, it's not clear to me how that would introduce a trade-off as discussed. Since, these two are very critical weaknesses, I decided to keep my score as is.

Dear Reviewer pRCH,

We sincerely thank you for the timely and thoughtful follow-up. We fully understand your concern regarding the relationship between our Bad-only training objectives and the observed trade-off across workflow conditions. Here we will make every effort to clarify your remaining concern .

  1. We use “trade-off” to describe the tension between utilizing helpful guidance and resisting misleading guidance, not explicit trade-off in training objective. An agent that prioritizes robustness may ignore workflows altogether, missing useful information. Conversely, greater reliance on workflows can improve performance with Good workflows, but also increase vulnerability to Bad workflows. More selective reliance can mitigate both risks. However, whether selective reliance improves both outcomes or entails a performance trade-off in complex real world also depends on several factors: task difficulty relative to the agent's capabilities, the severity and plausibility of misleading guidance, the agent's ability to retain relevant context, and even the ability agent reflect and revise its own CoT.
  2. Bad-only training can affect behavior beyond the training condition. Our objectives provide a possible explanation: SFT prescribes a correct trajectory under misleading guidance, whereas outcome-based RL optimizes task success without prescribing a specific trajectory. This allows RL to reinforce different strategies for handling external guidance, whose effects may generalize to Good workflows. Prior work [1] also provides an empirical precedent for such cross-condition transfer, showing that negative-only SFT can improve OOD reasoning performance. The same phenomenon is observed in our experiments.
  3. To further understand the behavioral changes underlying this phenomenon, we conduct a trajectory-level analysis on Mixed workflows. We evaluate 30 AIME 2026 problems, each with eight rollouts, and use an LLM judge (GPT-6.1-sol) to examine how models adopt or reject helpful and misleading steps. Among the pairs where RL succeeds but SFT fails, RL slightly more often adopts Good steps, correctly rejects Bad steps, and less often adopts misleading Bad steps. These trends suggest that RL does not simply increase or decrease workflow reliance, but may help the model better distinguish helpful from misleading guidance: retaining useful steps while rejecting misleading ones. This is consistent with our view that effective independent reasoning requires balancing workflow utilization with robustness.
Pairwise outcome# PairsModelGood adoption (↑)Incorrect Good rejection (↓)Misleading Bad adoption (↓)Correct Bad rejection (↑)
SFT correct, RL wrong24SFT10074
SFT correct, RL wrong24RL9065
RL correct, SFT wrong21SFT60111
RL correct, SFT wrong21RL7093

More broadly, we respectfully believe that systematically identifying and evaluating selective reliance on external guidance, and demonstrating this capability can be learned and generalized across contexts, are valuable contributions in their own right. There are many potential directions to explore, our analysis of training objectives, together with prior findings on negative-only training [1] and trajectory-level analysis, provides initial insights into this transfer. We hope our benchmark and finding can guide the community in designing more intelligent agents.

We sincerely appreciate your thoughtful feedback, which has helped us strengthen our work and clarify its contributions. We hope our responses have addressed your remaining concerns. We would be very grateful if you could consider whether our work merits a higher score. Please feel free to follow up and we will be happy to respond.

[1] Tian, Xueyun, et al. "Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization." arXiv preprint arXiv:2601.04992 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.