acceptodds
Under review as a conference paper at ICLR 2027

SGG-ReflAct: Sub-Goal Guided ReflAct with Structured Planning for Reliable Long-Horizon Reasoning

Abstract

Recent advances in reasoning backbones have empowered large language model (LLM) agents to tackle complex, multi-step tasks. However, as reasoning horizons grow, inconsistent internal beliefs induce intermediate errors that cause agents to drift from their goals. This limitation also persists in REFLACT, which reflects only on the end-goal at each step without explicitly considering intermediate subgoals. To address this problem, we propose SGG-ReflAct (Sub-Goal Guided ReflAct), a reasoning backbone that integrates sub-goals generated through a singlepath LLM planner into the reflection process. We further extend this framework to BeamSGG-ReflAct, which replaces the single-path planner with a beam searchbased LLM planner for structured plan exploration. We run experiments on ALFWorld, ScienceWorld, and Jericho with multiple LLM models. SGG-ReflAct outperforms REFLACT in nearly all settings, achieving best success rate gains of 14.9 percentage points on ALFWorld and 8.0 percentage points on ScienceWorld with Llama-3.1-8B-Instruct. Our experimental analysis shows that SGG-ReflAct reduces hallucinated actions and achieves its largest gains on procedurally ordered tasks. Furthermore, experimental results with BeamSGG-ReflAct show that the backbone’s effectiveness depends on plan quality: explicitly specifying the required operations recovers gains that plan searching alone cannot achieve. These results demonstrate that SGG-ReflAct offers a practical and highly effective reasoning backbone, enabling LLM agents to achieve reliable performance in complex, long-horizon tasks through easy integration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Reasoning backbone, long-horizon reasoning, self-reflection, planning, interactive decision making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UmAXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SGG-ReflAct, a sub-goal-guided reflection framework for long-horizon LLM agents. The method first uses an LLM planner to decompose a task into a sequence of sub-goals. During execution, the agent dynamically identifies the active sub-goal at each step and jointly performs reflection and action selection based on the current environment state and the current sub-goal. The authors further introduce an enhanced variant, BeamSGG-ReflAct, which uses beam search to improve sub-goal planning and achieve better task decomposition. Experiments on three benchmarks, ALFWorld, ScienceWorld, and Jericho, demonstrate that SGG-ReflAct outperforms existing baselines in most setups, with particularly noticeable improvements for LLMs with fewer parameters. Further ablation studies suggest that the performance gains primarily come from explicitly anchoring reflection to the active sub-goal. The authors find that the proposed method reduces the invalid-action rate, while more sophisticated planning search does not necessarily yield further improvements, highlighting the importance of plan quality for long-horizon agent performance.

Critical strengths
  1. The method is simple to implement, easy to adapt to different models and application scenarios, and introduces relatively low computational overhead. SGG-ReflAct is a runtime method that does not require additional model training or complex external modules. By simply combining sub-goal planning with sub-goal-anchored reflection, the method can effectively improve LLM performance on long-horizon agentic tasks. SGG-ReflAct only requires one additional LLM call for planning at the beginning of each task, without significantly increasing the per-turn runtime overhead. This makes the method highly practical and scalable.

  2. The experimental evaluation is relatively comprehensive and provides evidence that the method can effectively improve runtime performance. The authors conduct experiments across multiple benchmarks and LLMs of different sizes. The results show that the proposed method improves task success rates in most settings, with particularly noticeable gains for smaller models. Also, the authors conduct ablation studies to analyze the respective contributions of sub-goal planning and sub-goal-anchored reflection. They also investigate the effects of invalid-action rates and planning quality on overall performance. These analyses help explain why the method works, rather than merely demonstrating performance improvements.

Critical weaknesses
  1. The novelty of the proposed method is relatively limited, and its distinction from existing work is not sufficiently clear. The proposed method mainly combines sub-goal planning with step-wise reflection, replacing the final task goal with the active sub-goal as the target of reflection. However, similar ideas have already been explored in prior work. For example, Devil's Advocate (DEVIL’S ADVOCATE: Anticipatory Reflection for LLM Agents, Wang et al., 2024) also combines sub-task decomposition, pre-action reflection, and sub-task goal alignment to guide agent behavior. HiAgent (HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model, Hu et al., 2025) also investigates the use of sub-goals to organize long-horizon agent execution. Although the proposed method differs in its specific implementation, its overall technical novelty appears relatively limited. The authors should more clearly explain the key differences from these existing methods and provide more direct experimental comparisons to demonstrate the independent contribution of their approach.

  2. The experimental design and ablation studies have several limitations that weaken the support for the paper's main claims. The key ReflAct+Plan ablation study is only conducted on ScienceWorld with Llama-3.1-8B-Instruct, making it unclear whether the findings generalize to other models and benchmarks. This ablation also does not clearly separate the contributions of active sub-goal identification and sub-goal-anchored reflection, leaving the source of the performance improvements insufficiently understood. The experiments use only a single random seed without a statistical reliability analysis, even though some of the reported improvements are relatively small. The paper also relies heavily on invalid-action rate as a proxy for environment grounding. This metric does not directly measure whether the agent correctly understands the environment state or identifies the appropriate sub-goal, and the results do not show consistent improvements across all models. These limitations make it difficult to fully validate the proposed explanation for the performance gains.

Questions for authors
  1. How does SGG-ReflAct differ from existing methods listed in Critical Weaknesses section?

  2. Can the authors further disentangle the contributions of active sub-goal identification and sub-goal-anchored reflection through a more controlled ablation study?

  3. Can the authors provide stronger evidence of the statistical reliability of the reported improvements, given the use of a single random seed?

3 comments

We thank the reviewer for their constructive review. Tables R1–R5 are provided in the General Response.

W1/Q1. Difference from HiAgent and Devil's Advocate. While all three methods utilize sub-goals, they differ in how they utilize them at each step. We have re-examined both papers in full:

  • SGG-ReflAct generates N sub-goals once per episode. At every turn, the actor identifies the active sub-goal, reflects in one sentence on progress toward it, and selects an action conditioned on that reflection. It does not use an evaluator, memory module, or backtracking.
  • HiAgent uses sub-goals as units of working memory; once a sub-goal is achieved, it summarizes and hides the preceding trajectory. It does not require the agent to reflect on progress before acting.
  • Devil's Advocate prepares alternative actions before acting and, after acting, performs a YES/NO check to see if the outcome fits the current sub-task, backtracking (e.g., restoring web pages by URL) if it does not.
SGG-ReflActHiAgentDevil's Advocate
Role of sub-goalsReference of every reflectionUnits of memory compressionPlan to execute; regenerated on failure
Check against the current sub-goalFree-text progress assessment that conditions the actionAchievement check that triggers summarizationBinary YES/NO after acting, triggering rollback
Core contributionAnchoring reflection to the active sub-goalMemory compression and retrievalAnticipatory alternatives and backtracking

The presence of sub-goals and their specific application are distinct issues. In HiAgent's own ablation, sub-goals without memory compression ("w. TD") reach 40.0% on Tyreworld, positioned between the standard agent (10.0%) and the full method (60.0%). Since porting either system to our environments would necessitate original design choices, we isolated our contribution through the controlled comparison in Table R1. We will add both papers to the Related Work section.

W2. Scope of the ablation. This now covers seven settings (Table R1, General Response 2/5), including the two ALFWorld exceptions and their explanations.

Q2. Identification vs. anchoring. Because the identification and use of the active sub-goal occur within a single response, we cannot ablate one without the other. Identification is almost never wrong (regressions ≤ 0.68%, skips ≤ 1.63% of transitions in all nine settings; Table R6 in our response to rXbk). Since both ablation conditions utilize the same plan and the correct sub-goal is almost always identified, the difference comes from how the reflection uses that sub-goal. We will note the lack of a direct ablation as a limitation.

W2. The invalid-action rate is indirect. We agree and have added evidence categorized by failure types:

  • False completion (Robotouille, GPT-4o): 17 of ReflAct's 32 failures involve declaring an unfinished dish complete; SGG-ReflAct has 1, and 9 of the 13 episodes solved exclusively by SGG-ReflAct are of this type.
  • Procedural steps (runs of Table 1): on ALFWorld heat/clean/cool tasks, Llama-3.1-8B episodes performing the transformation increase from 12 to 40, and 19 of the 33 episodes solved exclusively by SGG-ReflAct were procedural failures under ReflAct. In ScienceWorld, failures involving repeated rejected commands drop from 52 to 33 (GPT-4o), accounting for 22 of the 30 episodes solved exclusively by SGG-ReflAct.
  • Where it helps and where it does not (Table R4): the gain appears on tasks with more prerequisite steps when each sub-goal covers at most approximately two such steps, though the net gain remains bounded in part by low-level action loops (General Response 4/5). Anchoring can also persist in the case of an incorrect formulation (ScienceWorld "boil lead": 146 turns of rejected variants).

Q3. Reliability of single runs. Following ReflAct's single-run protocol, and given the cost of multi-seed re-runs during the rebuttal period, we have added paired exact McNemar tests (Table R5): four of the six ALFWorld/ScienceWorld settings show significant results; the gain is not significant for GPT-4o on ALFWorld, which is at the performance ceiling, or for GPT-4o-mini on ScienceWorld, and Jericho cannot statistically separate the methods. Additional runs on ALFWorld GPT-4o-mini yielded 74.6–79.9% (ReflAct+Plan, 4 runs) and 70.1–76.1% (SGG-ReflAct, 3 runs). We will report these tests alongside Table 1 and will not characterize non-significant settings as improvements.

Revisions: HiAgent and Devil's Advocate in Related Work; the seven-setting ablation (Table 14, in the revised PDF); identification rates and failure-type analyses in the appendix; and the lack of a direct ablation, the persistence of anchoring, and low-level action loops in the Limitations.

The authors' response has addressed my concerns, and I will raise my score to 3.

Reviewer_ndHfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SGG-ReflAct, a sub-goal-guided extension of ReflAct designed to improve the reliability of LLM agents in long-horizon interactive tasks. The method first decomposes the original task into four ordered sub-goals and subsequently anchors each per-turn reflection to the currently active sub-goal rather than the final task objective. The authors additionally introduce BeamSGG-ReflAct, which uses self-evaluation-guided beam search to generate alternative sub-goal plans.

The approach is evaluated on ALFWorld, ScienceWorld, and Jericho using GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct. The reported results show improvements over ReAct, Plan-and-Act, and ReflAct in several settings, particularly with the smaller model. Additional analyses investigate invalid-action rates, the effect of reflection anchoring, and the relationship between plan quality and task success.

Critical strengths

S1. Simple and interpretable integration of planning and reflection.

The proposed method introduces a relatively straightforward modification to the ReflAct framework. Using intermediate sub-goals as reflection targets is intuitive, requires no additional model training, and can be integrated into existing LLM agent frameworks with limited changes to their execution process.

S2. Consistent improvements over established baselines in the evaluated environments.

Table 1 reports improvements across three benchmarks and three backbone models. In particular, SGG-ReflAct improves success rates over ReflAct by 14.9 percentage points on ALFWorld and 8.0 percentage points on ScienceWorld with Llama-3.1-8B-Instruct. These results suggest that the proposed strategy can benefit smaller models in structured interactive environments.

S3. Useful analyses of intermediate planning and action failures.

The authors investigate the contribution of sub-goal anchoring beyond simply including a plan in the context. The invalid-action analysis and qualitative examples also provide useful observations about how insufficiently specified plans can affect agent execution. The extensive appendix improves the transparency of the implementation and experimental setup.

Critical weaknesses

W1. The experimental evaluation does not adequately support the paper's central claim about reliable long-horizon reasoning.

My primary concern is the mismatch between the paper's broad research objective and its actual experimental scope. The evaluation is restricted to ALFWorld, ScienceWorld, and Jericho. Although these benchmarks are appropriate for controlled comparisons, their relatively constrained interaction patterns and predefined task structures limit the evidence for general long-horizon reasoning. The Jericho evaluation contains only 20 games, further limiting the conclusions that can be drawn from relatively small differences in success rates.

More importantly, the paper does not systematically establish how the proposed method behaves as the reasoning horizon increases. Dividing trajectories into early, middle, and late stages does not directly demonstrate that the method becomes more effective as task length or dependency complexity grows.

Furthermore, the horizons actually exercised are short, and ALFWorld in particular is not a representative long-horizon benchmark by current standards. Table 10 shows that on episodes solved by both agents, mean steps to success are only about 9–14 on ALFWorld and ScienceWorld (and the Jericho overlap is at most 7 episodes per model), while episode step budgets reach 30–120. In contrast, established long-horizon agent benchmarks—such as NetHack in BALROG or AppWorld—routinely require hundreds of environment steps or dozens of cross-application API calls per successful episode. The tasks in which the method demonstrably helps are therefore those whose successful trajectories occupy a small fraction of the available horizon, and the paper never reports success rates stratified by required plan length. The "long-horizon" framing is thus supported mainly by tasks that fail at long horizons rather than by demonstrated competence at them.

To support its central claim, the paper should include a systematic analysis of success rates and error accumulation across different task lengths or dependency depths, preferably complemented by evaluation on a more open-ended agent benchmark such as WebArena.

W2. The methodological contribution is incremental, and its generality remains insufficiently established.

The main contribution consists of combining upfront sub-goal decomposition with the existing ReflAct framework and redirecting each reflection toward the active sub-goal. Both hierarchical planning and reflection-based action selection are established techniques. The paper provides an intuitive integration of these ideas, but the methodological advancement appears limited.

Moreover, the framework always generates exactly four sub-goals. Appendix B indicates that this choice was informed by an analysis of ScienceWorld gold action sequences, weighted by the number of evaluation episodes. Together with the benchmark-specific planner instructions, this raises concerns about how much of the improvement depends on prior knowledge of the evaluation environments. The paper's own results hint at this brittleness at both ends of the spectrum: on two-step Lifespan tasks the planner still emits four sub-goals and SGG-ReflAct underperforms ReflAct (53.3% vs. 66.7%), while on genuinely long-horizon Grow tasks no agent exceeds 5% success—regimes the fixed N = 4 design is not equipped to handle.

The authors should provide sensitivity experiments with different numbers of sub-goals and evaluate less specialized planning prompts or previously unseen task structures. It would also be helpful to clarify whether evaluation-set gold trajectories were consulted before the planning configuration was finalized.

W3. The experimental setup is insufficient to establish robustness across contemporary LLM capabilities.

The model selection is limited to GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct. For a submission targeting ICLR 2027, this leaves an important gap in understanding whether the proposed approach remains useful with more capable contemporary open-weight models.

The method may primarily compensate for planning weaknesses of smaller or older models. Without stronger contemporary baselines, it is difficult to determine whether the reported improvements represent a broadly useful reasoning mechanism.

Furthermore, the experiments use a single fixed random seed. The standard deviations reported in Table 1 measure variation across task categories rather than variation across independent trials. Several improvements are also small; for example, the GPT-4o success rate on ALFWorld increases by only 0.7 percentage points over ReflAct.

The authors should evaluate a stronger contemporary model, such as Qwen3.5-27B, and provide appropriate paired statistical uncertainty estimates or repeated evaluations under varying environment conditions.

The paper also does not situate its results within the current performance landscape. Recent work reports success rates of 94–99% on ALFWorld using 1.5B–7B models fine-tuned with reinforcement learning (e.g., RSIPO: 98.6% with Qwen2.5-7B), with multiple task categories already at 100%. The best result reported here is 90.3% using GPT-4o with a prompt-only method. The paper neither cites nor discusses these higher-performing baselines, leaving readers unable to judge where a prompt-level improvement stands relative to what is already achievable. Without that comparison, the practical relevance of the reported gains is unclear.

W4. The causal analysis is too narrow to justify the general claims, while the search extension lacks a convincing computational benefit.

The central ablation separating plan inclusion from sub-goal reflection anchoring (Table 13) is conducted only on ScienceWorld with Llama-3.1-8B-Instruct. This is useful evidence for one setting, but is insufficient to establish that reflection anchoring explains the performance gains across all evaluated models and environments.

The invalid-action analysis also reveals less consistent improvements than the general framing suggests. For GPT-4o-mini, the overall invalid-action rate of SGG-ReflAct is almost unchanged relative to ReflAct (32.6% versus 32.7%), while the late-episode invalid-action rate is higher (45.5% versus 39.0%).

Furthermore, BeamSGG-ReflAct requires up to 42 planning calls per episode, yet its success rate is below that of the single-path method in all nine model-benchmark combinations. The experimental evidence therefore does not establish a convincing benefit from the more elaborate search procedure.

The authors should extend the central ablation across additional models and environments and provide a systematic performance-versus-computation comparison, including token usage and inference latency. Such experiments are important for determining whether the proposed framework provides a meaningful advantage beyond prompt-based planning heuristics.

Questions for authors

Q1. Evidence for long-horizon reasoning. Can the authors provide a systematic breakdown of performance by task length or sub-goal dependency depth? In particular, do the improvements over ReflAct become larger as the required action sequence grows longer, or are they primarily concentrated in tasks with simple, predefined procedures?

Q2. Generalization beyond the evaluated model and task configurations. How sensitive is the method to the fixed four-sub-goal design and benchmark-specific planner instructions? Can the authors provide results with a stronger contemporary open-weight model, such as Qwen3.5-27B, to establish whether sub-goal anchoring remains beneficial when the backbone already has stronger planning capabilities?

Q3. Robustness and computational efficiency. Can the authors extend the ReflAct+Plan ablation to additional model-benchmark combinations and report paired uncertainty estimates for the main results? Additionally, what are the actual token and latency costs of SGG-ReflAct and BeamSGG-ReflAct, especially considering that beam search consistently underperforms the single-path method?

Q4. Evaluation protocol. Appendix B describes using ScienceWorld gold action sequences to justify the fixed number of sub-goals. Were these gold trajectories examined before finalizing the method? Also, how sensitive are the ScienceWorld conclusions to the success threshold of 0.7 used in the evaluation?

3 comments

We thank the reviewer for their rigorous review. We agree with W1 and W4: the submission lacked length-based evidence and an ablation beyond one setting. Tables R1–R5 are provided in the General Response.

W1/Q1. Long-horizon evidence. We adopted the term "long-horizon" as it is used for these benchmarks (BEACON evaluates on "three long-horizon benchmarks: ALFWorld, ScienceWorld and WebShop"): the difficulty stems from a chain of implicit intermediate goals rather than the raw step count. We agree this requires evidence by length and will define the term in the Introduction.

On Table 10 (Table 11 in the revision). This table includes only episodes solved by both agents, which are short by construction; failed episodes are approximately twice as long (e.g., for Llama-3.1-8B in ScienceWorld, SGG-ReflAct takes 13.3 vs. 22.2 steps, and ReflAct takes 10.6 vs. 23.2).

Table R7. SR (%) by optimal-path length L* (BEACON strata: ≤ 4 Short, 5–7 Medium, > 7 Long), ReflAct → SGG-ReflAct.

ALFWorldShort (42)Medium (75)Long (17)
GPT-4o97.6 → 97.688.0 → 89.376.5 → 76.5
GPT-4o-mini73.8 → 83.366.7 → 77.317.6 → 23.5
Llama-3.1-8B71.4 → 61.913.3 → 48.047.1 → 35.3

On ScienceWorld, 183 of 211 episodes are Long, where SGG-ReflAct leads for all three models (GPT-4o 49.7 → 60.1, GPT-4o-mini 35.5 → 41.0, Llama-3.1-8B 20.2 → 28.4). On ALFWorld Medium+Long, the paired tests yield 31 : 7 (p<0.001) for Llama-3.1-8B and 13 : 4 (p=0.049) for GPT-4o-mini.

Answer to Q1. The gain is not concentrated in short procedures: it is minimal or negative on the shortest tasks and appears on tasks with more prerequisite steps (Table R4). On Robotouille, GPT-4o's gain occurs in tasks of 10–24 optimal steps (4/24 → 16/24), which are longer than the 9–14-step trajectories noted by the reviewer. On the longest tasks (36–63 steps), N=4 bundles 2–3 steps per sub-goal and shows no gain; raising N to 8 and 12 yields 5 and 6 successes against ReflAct's 4 (General Response 4/5; single run).

Scope. NetHack, AppWorld, and WebArena add exploration or web/API skills in addition to task length. We chose Robotouille because its dependencies are explicit and each step can be verified through the environment state, isolating the effect of anchoring to sub-goals.

Error accumulation. Within an episode, ReflAct's invalid-action rate increases to 71.4% in the final third (Llama-3.1-8B, ScienceWorld), compared to 49.7% for SGG-ReflAct.

W2/Q2. Incremental contribution, N=4, planner instructions.

Contribution. Simply including a plan in context while maintaining the end-goal reflection provides minimal improvement (Table R1: −20.0 to +1.9 pp over ReflAct in five of seven settings). In contrast, SGG-ReflAct—which differs only by anchoring each reflection to the active sub-goal—achieves gains of +4.7 to +10.5 pp in those same settings.

N=4 and gold trajectories. Gold action sequences were used only once to compute the forced-order chain length for each ScienceWorld task type (mean 3.79, median and mode 4); no gold trajectories or text derived from them are included in any prompt. We will explicitly state this in Appendix B.

Table R8. Sensitivity to N (successes; planner example rewritten to match N; ReflAct in the same configuration). Parentheses give the mean number of sub-goals the planner actually produced.

SettingEpisodesN=4Other NReflAct
ScienceWorld 2-step tasks, Llama-3.1-8B481717 (N=2; 2.6)16
ScienceWorld 8-step tasks (Chemistry mix), Llama-3.1-8B800 (N=8; 8.8)0
Robotouille, GPT-4o-mini (earlier planner rules v1 / v5)405 / 44 / 3 (N=8)6
Robotouille long tasks, GPT-4o1635 (N=8; 8.0), 6 (N=12; 14.9)4

On short tasks, we see no evidence that N drives the outcome (paired N=2 vs. N=4: 7 : 7). On the eight-step Chemistry-mix tasks, every agent, including ReflAct, fails, making the choice of N irrelevant. For long Robotouille tasks, increasing N toward the task's stage count improves performance. However, the planner does not follow N exactly (producing 2.6 sub-goals for N=2, 8.8 for N=8, and 14.9 for N=12). Therefore, N=4 is not tuned to the evaluation environments; rather, N is a design parameter that should scale with the number of prerequisite steps, a point we will clarify in the paper.

Benchmark-specific instructions. Every agent, including ReflAct, receives benchmark-specific prompts and examples. The risk identified by the reviewer did manifest on Robotouille: without restating the environment description in the planner prompt, plan quality dropped sharply (GPT-4o-mini SGG-ReflAct 1/40 vs. 8/40), consistent with Section 5.2. We will document this dependence in the Limitations.

W3. RL fine-tuned agents. Methods such as RSIPO update the model weights with reinforcement learning on ALFWorld; RSIPO’s own Table 1 reports Qwen2.5-7B-Instruct at 12.5% before training and 98.6% after it. SGG-ReflAct keeps the weights frozen and changes only the prompt and control flow, so the two lines of work are complementary rather than competing: a fine-tuned policy can also run inside the sub-goal–anchored reflection loop. We will add this line of work to Related Work and state the distinction explicitly.

W4/Q3. Ablation, invalid actions, beam cost.

Ablation. This now covers seven settings (Table R1). We report the two ALFWorld exceptions and avoid claiming that anchoring explains the gain in all cases; General Response 2/5 shows that while anchoring still reduces procedural failures to one episode in those settings, search failures account for half or more of the remaining errors.

Invalid actions for GPT-4o-mini. The late-episode rate is indeed higher with SGG-ReflAct (45.5% vs. 39.0%). In the ScienceWorld "boil lead" task, SGG-ReflAct follows its sub-goal's wording ("pick up paper cup containing lead") and, anchored to that sub-goal, repeats rejected variants for 146 turns, whereas ReflAct succeeds in 25. This suggests that anchoring can persist even with an incorrect formulation; we will add this to the Discussion. Additionally, other metrics demonstrate the intended effect more directly: on Robotouille with GPT-4o, ReflAct results in 17 failures where an unfinished dish is declared complete, compared to only 1 for SGG-ReflAct.

Beam cost. SGG-ReflAct requires only one planning call, compared to up to 42 for BeamSGG-ReflAct, with equal per-turn costs. Since we did not log exact token counts, we report call counts; the beam approach is not cost-effective with these models.

Q4. Threshold. The 0.7 threshold follows ReflAct's success criterion, which was adopted because the built-in ScienceWorld flags are triggered at low progress levels. The ranking remains consistent from 0.5 to 1.0 (SR %, ReflAct / SGG-ReflAct):

Table R9. Threshold sweep on ScienceWorld.

ThresholdGPT-4oGPT-4o-miniLlama-3.1-8B
0.568.2 / 76.856.4 / 62.137.0 / 46.0
0.7 (paper)55.5 / 64.040.3 / 45.025.6 / 33.6
0.937.9 / 42.220.4 / 26.518.0 / 26.5
1.037.9 / 42.220.4 / 26.518.0 / 26.5

Revisions: restricting the claim to tasks requiring strict prerequisite ordering; adding Robotouille (Table 1, Appendix F) and the seven-setting ablation (Table 14) to the revised PDF; including length analyses, paired tests, N sensitivity, and the threshold sweep in the appendix; defining "long-horizon"; and addressing the dependence on planner instructions, the persistence of anchoring on incorrect formulations, missing preconditions, low-level action loops, and beam cost in the Limitations.

Reviewer_rXbkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SGG-ReflAct, which augments ReflAct with an initial, sub-goal plan generated by a planner module, and directs each subsequent reflection toward the currently active sub-goal. Its central hypothesis is that intermediate objectives provide more actionable guidance than reflection against a distant end-goal. The authors also propose BeamSGG-ReflAct, which replaces the single-call planner with self-evaluation-guided beam search. Evaluation covers three models (GPT-4o, GPT-4o-mini, Llama-3.1-8B-Instruct) on ALFWorld, ScienceWorld, and Jericho. SGG-ReflAct achieves higher reported success rates than ReflAct, and nearly all other baselines in all nine model–benchmark combinations. An ablation supports explicitly directing reflection toward active sub-goals rather than merely including a plan in context. Additional analyses of plan content reveal that explicit procedural guidance of the planner to include operations to achieve the goal can improve results.

Critical strengths

SGG-ReflAct is a very simple approach that requires one additional planning call per episode and no model training or extra per-turn calls. The substantial improvements, especially on smaller models, together with gains across three environments, validate the effectiveness of the approach.

Critical weaknesses

The evaluations don’t consider some key baselines like ToT, MAP, and also don’t reliably establish whether the major gains come on tasks that are really long horizon. Without this, the main claim that subgoal guided reflection helps in long horizon reasoning is not justified. Generalizability to more realistic task environments like WebArena remains to be seen.

Questions for authors
  1. Why are baselines like ToT/GoT excluded from evaluation? Also MAP [1] is a relevant approach worth discussing and comparing, which also generates subgoals at the beginning and pursues them sequentially.

  2. How often does the agent select the wrong active subgoal?

  3. Does SGG-ReflAct really improve for long horizon tasks? A breakdown of accuracy vs max steps would be helpful.

  4. Why does BeamSGG-ReflAct consistently underperform SGG-ReflAct when intuitively it should be able to search and select a good plan? Have the authors tried tuning the search parameters or other search techniques (like DFS)?

[1] - Webb, T., Mondal, S.S. and Momennejad, I., 2025. A brain-inspired agentic architecture to improve planning with LLMs. Nature Communications, 16(1), p.8633.

3 comments

We thank the reviewer for their careful reading. Tables R1–R5 are provided in the General Response.

Q1. ToT/GoT and MAP. ToT and GoT search over thoughts for static problems (e.g., Game of 24, sorting); they receive no observations and lack an active sub-goal or per-turn reflection, so applying them to ALFWorld would require designing a new agent. MAP is more closely related and will be added to the Related Work section. In the reported experiments, its decomposer generates a single intermediate goal, its gains derive from tree search over Actor/Predictor/Evaluator modules, and it utilizes approximately 148.6 LLM calls per Tower of Hanoi problem, whereas SGG-ReflAct adds only one planning call per episode. Our claim is not that sub-goal decomposition is novel, but that anchoring each reflection to the active sub-goal improves the reflection backbone at minimal cost; Table R1 tests this directly.

Q2. How often is the wrong sub-goal selected? This occurs rarely across all nine settings:

Table R6. Regressions to an earlier sub-goal / skips of two or more, as a share of sub-goal transitions.

Llama-3.1-8BGPT-4o-miniGPT-4o
ScienceWorld0.03% / 0.32%0.19% / 0.85%0.68% / 1.63%
ALFWorld0.33% / 0.90%0.21% / 0.21%0.12% / 0.12%
Jericho0.00% / 0.00%0.00% / 0.21%0.00% / 0.44%

Between 3% and 31% of turns could not be matched to a sub-goal; most of these are either idle turns after task completion or turns within a repeated-failure loop where the agent stops restating its sub-goal. We will report both caveats.

Q3. Is the gain really on long-horizon tasks? We performed the requested breakdown:

  • Dependency depth: On ScienceWorld, the gain is +2.4 pp for forced-order chains of ≤ 3 steps and +10.2 pp for ≥ 4, across all models (Table R4).
  • Required plan length (BEACON strata): On ALFWorld, gains appear on Medium rather than Short tasks (Llama-3.1-8B 13.3 → 48.0), and on ScienceWorld Long tasks, SGG-ReflAct leads for all three models (Table R7 in our response to ndHf).
  • Robotouille (optimal 10–63 steps): For GPT-4o, SGG-ReflAct achieves 19 vs. 8 (p=0.007), with the largest gains on medium tasks (5/8 vs. 0/8). On the longest tasks, N=4 shows no gain because one sub-goal bundles 2–3 steps; with N=12, SGG-ReflAct solves 6/16 vs. 4/16 (single run).

Thus, the gain is concentrated on tasks requiring many prerequisite steps, provided the sub-goals represent them with sufficient granularity; it does not increase with task length alone.

Q4. Why does BeamSGG-ReflAct underperform? Its bottleneck is scoring rather than search width. The beam constructs the plan one sub-goal at a time, scoring each candidate primarily against those already selected, which causes information about the overall task to be lost. For example, in "make violet paint" (Llama-3.1-8B), the single-call planner writes "mix the materials" and succeeds in 9/9 episodes, whereas the beam writes "focus on the pigment" and collects only one paint (0/9). On Robotouille with GPT-4o-mini, the beam plan includes the second dish in 2 of 12 two-dish tasks (SGG-ReflAct: 10). Since a greedy search with a similar budget yields similar results (27.0% vs. 26.5%, Appendix J of the revision), we did not expect DFS to improve performance under the same scoring. With GPT-4o on Robotouille, the beam performance is close to SGG-ReflAct (17 vs. 19). Because the beam requires up to 42 planning calls compared to a single call, we will state that it is not cost-effective for these models.

WebArena. We maintained ReflAct's evaluation suite for direct comparison and added Robotouille to test longer procedures; web environments remain outside our current scope, which we will explicitly state in the Limitations.

Revisions: Robotouille (Table 1, Appendix F), the length/depth analyses, the N ablation, wrong-sub-goal rates, the beam failure analysis and cost, MAP in Related Work, and the scope statement.

Thank you for the response. The rebuttal has addressed some of my concerns. I will keep my rating.

Comments on the paper

5 comments

We thank the three reviewers for their careful and constructive feedback. All three recognized the simplicity of SGG-ReflAct and its advantages on smaller models, while identifying precisely where additional evidence was needed. In response, we conducted new experiments and analyses, which we summarize in this general response (Tables R1–R5) and address in detail in our individual responses.

Clarification on our original evaluation setup. Before presenting the new results, we clarify the rationale for our original experimental setup. To ensure performance improvements are fairly attributed to our method, we followed the environments and protocols of ReflAct, the framework we extend:

  • Benchmarks and splits: We evaluated on ALFWorld (134 unseen), ScienceWorld (211), and Jericho (20)—the same three environments as ReflAct—using the standard splits from ReflAct/MPO and AgentBoard. Recent long-horizon agent studies use the same benchmark family; for instance, BEACON (Wang et al., 2026) evaluates on ALFWorld, ScienceWorld, and WebShop as "three long-horizon benchmarks."
  • Baselines and success criterion: We included all ReflAct baselines (NoThinking, ReAct, Plan-and-Act, and ReflAct) and judged success based on the progress reward, as ReflAct does, because the built-in success flags in ScienceWorld and Jericho are unreliable.
  • Deterministic single runs: Following ReflAct, Plan-and-Act, MPO, and 9 of the 11 recent planning and reflection agent papers we surveyed, we ran each setting once at temperature 0; only one of these papers (PlanU) reported variance across seeds.

Summary of new experiments. In response to the reviewers' shared feedback, we addressed three questions: (1) whether the gain comes from anchoring the reflection to the active sub-goal rather than merely having a plan, (2) evidence on longer-horizon tasks, (3) across varying numbers of sub-goals N, and (4) the reliability of single-run evaluations. Each experiment below is followed by its result and analysis: (a) sub-goal anchoring vs. plan availability, (b) a longer benchmark, Robotouille, (c) the number of sub-goals N on long tasks, and (d) paired tests on the original runs due to compute and time constraints on full multi-seed re-runs during the rebuttal period. Experiments (a) and (b) are already incorporated into the revised PDF (Table 1, Table 14, and Appendix F); the remaining analyses will be added to the appendix.

(a) Sub-goal anchoring vs. plan availability (seven settings, new runs). To isolate the effect of sub-goal anchoring from the mere presence of a plan in context, we evaluated ReflAct+Plan, a variant that receives the exact same sub-goal plan as SGG-ReflAct but continues to reflect against the end goal. While this ablation previously only covered ScienceWorld with Llama-3.1-8B, we have now extended it across all three benchmarks and model backbones where the comparison is informative.

Table R1. Ablation (SR %). Last two columns: gain over ReflAct (exact McNemar p against ReflAct).

SettingReflAct+PlanSGG-ReflAct+Plan vs. ReflActSGG-ReflAct vs. ReflAct
ScienceWorld, Llama-3.1-8B25.627.533.6+1.9 (0.67)+8.0 (0.036)
ScienceWorld, GPT-4o55.556.964.0+1.4 (0.71)+8.5 (0.008)
ScienceWorld, GPT-4o-mini40.338.445.0−1.9 (0.59)+4.7 (0.16)
ALFWorld, Llama-3.1-8B35.835.146.3−0.7 (1.00)+10.5 (0.054)
Jericho, GPT-4o35.015.040.0−20.0 (0.13)+5.0 (1.00)
ALFWorld, GPT-4o89.691.090.3+1.4 (0.73)+0.7 (1.00)
ALFWorld, GPT-4o-mini62.774.676.1+11.9 (0.002)+13.4 (0.002)

Result. In five of the seven settings, providing the plan alone alters the success rate (SR) by −20.0 to +1.9 percentage points (pp) relative to ReflAct, whereas SGG-ReflAct achieves gains of +4.7 to +10.5 pp. On ALFWorld with GPT-4o and GPT-4o-mini, providing the plan alone yields gains comparable to SGG-ReflAct (+1.4 vs. +0.7 pp and +11.9 vs. +13.4 pp, respectively). (The ALFWorld GPT-4o-mini SGG-ReflAct entry is a rerun, and ALFWorld Llama-3.1-8B uses a single 16k-context configuration for all three columns.)

Analysis: the gain comes from anchoring, not from the plan. Merely exposing the agent to a plan is insufficient; consistent gains emerge only when reflection is explicitly anchored to the active sub-goal. The two ALFWorld exceptions do not indicate a failure of anchoring, but rather that there was little left for anchoring to address. On ALFWorld, anchoring targets procedural errors, such as skipping or failing a required heat, clean, or cool step. Failures where the agent fails to locate the target object are search problems that neither the plan nor anchoring is designed to address. For GPT-4o-mini, adding the plan reduces procedural failures from 15 (ReflAct) to 8, and anchoring further reduces them to 1; however, search failures increase from 13 to 16, resulting in a net gain of only two episodes (100 → 102 of 134). For GPT-4o, procedural failures are already at 1 under ReflAct and remain at 1, with 9 of its 12–13 remaining failures being search-related. In contrast, Llama-3.1-8B still has 29 procedural failures after the plan is added, but anchoring reduces them to 16 and achieves 15 additional successes (47 → 62). This divergence may reflect that stronger models such as GPT-4o and GPT-4o-mini are better able to follow a high-level plan on their own, whereas smaller models appear to benefit more from explicit sub-goal anchoring. In practical Physical AI settings—where compact on-board models are often preferred over large server-side LLMs—enhancing the reasoning stability of smaller models can therefore be particularly valuable.

(b) A longer benchmark: Robotouille (new runs). To evaluate performance on longer-horizon procedures, we added Robotouille (Gonzalez-Pumariega et al., ICLR 2025), a cooking benchmark requiring ingredients to be prepared (cut or cooked) and stacked in a fixed order. Optimal plans typically require 10–63 actions, with the longest tasks involving two dishes on separate tables. We conducted 40 episodes (four layouts for each of the ten synchronous tasks) using the official step limit (1.5 × optimal), the three models from the paper, and all six agents. All agents receive the official environment description; since the planner only sees the goal and the initial observation, its prompt restates this description as kitchen rules (Appendix F).

Table R2. Robotouille successes (N = 4, episodes = 40), and by task length for ReflAct → SGG-ReflAct.

ModelNoThinkingReActPlan-and-ActReflActSGG-ReflActBeamSGG-ReflAct
GPT-4o2121081917
GPT-4o-mini264684
Llama-3.1-8B011111
ReflAct → SGG-ReflActShort (16 episodes, 10–15 steps)Medium (8, 23–24)Long (16, 36–63)
GPT-4o4 → 110 → 54 → 3
GPT-4o-mini4 → 42 → 20 → 2

Result. For GPT-4o, SGG-ReflAct solves 19 of 40 episodes compared to 8 for ReflAct (13 episodes solved only by SGG-ReflAct vs. 2 only by ReflAct, p=0.007), with gains observed on short and medium tasks. GPT-4o-mini gains two episodes (p=0.69), while Llama-3.1-8B solves at most one episode with any agent. On long tasks, the default N=4 shows no gain; experiment (c) investigates this.

Analysis: anchoring prevents false completion. Error logs reveal the source of the gains. With GPT-4o, 17 of ReflAct's 32 failures involve declaring an unfinished dish complete, typically due to an incorrect stacking order; SGG-ReflAct has only one such failure, and 9 of the 13 episodes it alone solves are of this type. Conversely, Plan-and-Act, which follows an ordered plan without anchored reflection, exhibits 16 such failures. Therefore, a plan alone does not prevent false completion; rather, reflecting against the remaining sub-goals does.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.