acceptodds
Under review as a conference paper at ICLR 2027

OverForge: Reasoning Through Strategies and Tactics Helps Cooperative Lifelong Adaptation

15 views

Abstract

Cooperative language-model agents must coordinate over long horizons and adapt to changing environments and to partners with unfamiliar conventions, yet existing agents map observations to actions without separating persistent coordination strategies from their tactical execution. We introduce OverForge, a training-free hierarchical architecture that separates strategic reasoning over roles and divisions of labour from tactical reasoning over actions within each agent’s private, partner- conditioned world model. A metacognitive Prefrontal Cortex Module couples the two levels by forming strategy–action branches, imagining their consequences with a forward model, and committing when confident. In OvercookedV2, OverForge delivers 7 soups in a connected kitchen versus 3 for each flat LLM baseline, retains agreed roles, and adopts roles proposed by unfamiliar partners. Ablations and a fixed-strategy probe show that persistent strategies guide tactical adaptation while each reasoning level contributes to coordination. Memory restarts show that cross-episode partner knowledge supports task performance and partner prediction, linking the hierarchy to continual adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multi-agent cooperation, large language models, hierarchical reasoning, world models, metacognitive control, Theory of Mind, lifelong adaptation, zero-shot coordination, in-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jUkZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OverForge, a training-free hierarchical LLM agent that separates coordination strategies from tactical actions and uses partner-conditioned rollouts for decision-making. Experiments in OvercookedV2 evaluate coordination, partner adaptation, and cross-episode memory.

Critical strengths

S1. The strategy-tactic hierarchy provides an intuitive and interpretable framework for cooperative planning. S2. The paper includes component ablations and detailed behavioral analyses, showing improvements over flat LLM baselines in one kitchen layout.

Critical weaknesses

W1. Limited generalizability. Experiments are restricted to OvercookedV2, a single LLM backbone, and only five episodes per condition. Performance gains are also inconsistent across layouts, while the method relies on task-specific prompts. W2. Insufficient evidence for lifelong adaptation. OverForge largely builds on MindForge's memory mechanisms, while the memory-restart experiments do not isolate the hierarchy's contribution to cross-episode adaptation. W3. Inadequate baseline comparisons. The paper lacks comparisons with existing hierarchical agents. Existing baselines also differ in components and inference costs, with OverForge requiring roughly three times as many LLM calls as MindForge, making OverForge's advantages unclear.

Questions for authors

Q1. Can the authors provide results on other benchmarks or LLM backbones, along with independent runs to assess the robustness and generalizability of the improvements? Q2. Can the authors compare OverForge with its non-hierarchical variant under the same cross-episode memory setting? Q3. Can the authors compare against existing hierarchical methods while controlling for memory, partner modeling, and inference budget?

Reviewer_iCBwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents OverForge, a prompting-based cooperative multi-agent framework built on MindForge. It separates high-level role planning from action selection, uses an LLM to predict and score action consequences, and adjusts rollout depth through an entropy-based stopping rule. Experiments use a frozen Qwen-3.5-27B model in OvercookedV2.

Critical strengths

The ablations suggest that role planning and lookahead improve soup delivery in the connected kitchen. The paper also examines role persistence and coordination failures, providing some insight into how the controller affects cooperation.

Critical weaknesses
  1. The method’s advance over existing planning methods is not clear, and several claims need stronger support. It combines role planning, lookahead and stopping mechanisms within MindForge, but the JEPA and prefrontal analogies do not explain what is technically new. The attribution of split-kitchen failures to missing environmental information is not tested through a controlled comparison. The claim that the gains are independent of model scale is also difficult to assess from experiments with a single model.

  2. The experiments are limited to OvercookedV2 and a single Qwen-3.5-27B model. Across the two tested kitchen layouts, the method delivers more soups than the flat LLM baselines in the connected kitchen, but not in the split kitchen. It is therefore unclear whether these findings extend to other cooperative tasks.

Questions for authors

The method improves soup deliveries over the flat LLM baselines in the connected kitchen, but not in the split kitchen. What evidence supports its usefulness in other cooperative tasks?

Reviewer_vEYrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OverForge proposes a training-free, hierarchical architecture for cooperative language-model agents in partially observable environments (OvercookedV2). The core idea is to separate strategic reasoning (persistent coordination policies—roles, divisions of labour) from tactical reasoning (state-dependent primitive actions) inside each agent's private, partner-conditioned world model. A Prefrontal Cortex Module (PCM) couples the two levels: it proposes (strategy, first-action) branches, imagines their consequences with a forward model, judges them via pairwise comparison (immediate value) and rollout (trajectory value), and commits only when a metacognitive confidence threshold is reached; otherwise it deepens deliberation on a nucleus of plausible branches.

Critical strengths
  1. Clear and well-motivated conceptual distinction. Separating persistent coordination strategies from tactical execution is intuitively right and connects to cognitive-science work on prefrontal hierarchical control. The paper operationalizes this distinction concretely (strategy = role/division of labour without coordinates or primitives; tactic = primitive action conditioned on state) and tests it with a pinned-strategy probe.
  2. Honest reporting of negative and mixed results. The split-room failures are attributed to a missing constraint channel rather than hidden; the cross-play execution gap (OverForge agrees but fails interactions) is reported; memory restart helps in the connected room but not consistently in the split room; IPPO collapses out of distribution and the paper explicitly says this should not be read as a statement about RL in general. This honesty is a strength.
  3. Genuine architectural contribution beyond flat prompting. The PCM is not just “ask the LLM twice.” It proposes branches at the first action, scores immediate value by pairwise comparison, rolls out partner-conditioned futures with a forward model, computes a branch posterior and metacognitive confidence, and deepens only on a nucleus.
Critical weaknesses
  1. Cross-play execution gap is acknowledged but not resolved. OverForge adopts unfamiliar partners’ roles (counters only 4–5% of proposals) but fails 0.80 of its interactions in cross-play, versus 0.29 and 0.14 for MindForge and MindForge-C (Table 7). The paper calls this “the gap between words and deeds,” which is honest, but it undercuts the “adopts roles proposed by unfamiliar partners” claim in the abstract: the agent agrees in language but does not execute reliably.
  2. LLM-as-a-judge concerns are acknowledged but not fully addressed. The PCM relies on LLM judges for pairwise action comparison and state value. The paper fixes the judge model, prompts, and scale, uses comparative judgements, and reports unparseable-judge rates (19% connected, 21% split, concentrated at rollout depth). But it does not report judge–human agreement, judge–outcome correlation, or sensitivity to the judge model. Given that the method’s decisions are mediated by these judges, this is a significant gap.
  3. Some claims in the abstract/introduction are stronger than the evidence. “OverForge delivers 7 soups in a connected kitchen versus 3 for each flat LLM baseline” — this is a connected-room, five-episode result; the split room is much weaker. The abstract should qualify the room and the variance. “retains agreed roles, and adopts roles proposed by unfamiliar partners” — the cross-play execution gap means adoption is partial and execution is poor. “linking the hierarchy to continual adaptation” — the memory-restart evidence is suggestive but mixed across rooms.
Questions for authors
  1. OverForge uses ~3× more LLM calls per agent-step than MindForge (40.2 vs. 12.9 in the connected room). Can you add a compute-matched flat baseline—e.g., MindForge with self-consistency over multiple samples, a larger comparison budget, or a flat agent allowed the same total LLM calls—to separate the effect of hierarchical decomposition from the effect of more inference?
  2. In the pinned-strategy probe, the pinned seat bypasses the strategic call. Can you isolate the effect of strategy persistence from the effect of removing strategic inference (e.g., a pinned strategy that still runs the strategic call but ignores its output, or a free seat with a matched call budget)?
  3. How many independent seeds/runs underlie each reported mean ± sd? Five episodes per condition is few. Can you report confidence intervals or bootstrap standard errors for the headline comparisons (deliveries, success rate, partner-intent accuracy, failed-interaction rate)?
  4. The ablation results are directionally inconsistent across rooms (no-ranking delivers 1.0 vs. 1.4 in the connected room but 0.8 vs. 0.2 in the split room). Can you explain this interaction and report whether it is stable across seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.