acceptodds
Under review as a conference paper at ICLR 2027

CB-Agent: Concept-Based In-Context Learning for Sequential Action Selection

Abstract

In-context learning (ICL) allows large language models to use demonstrations without updating their parameters. For interactive agents, demonstrations are trajectories of reasoning, actions, and observations that illustrate how to complete a task. Reusing demonstrations in new situations is challenging: directly inputting full trajectories consumes limited prompt space and constrains the overall effectiveness, while natural-language summaries require the agent to translate general guidance into specific action choices. Therefore, we study whether expert action choices can directly supervise a reusable scoring rule. We propose CB-Agent, which extends concept-based ICL to sequential action selection. We extract environment-action decisions from expert trajectories and fit a concept vector by maximum likelihood to model the conditional distribution over candidate actions given the context. This model combines the concept with frozen language-model features and priors. During ReAct interaction, the agent uses this distribution to select environment actions without including demonstrations in the action-scoring context. Across TextWorld, ALFWorld, and ScienceWorld, CB-Agent achieves the highest cross-benchmark average among the evaluated methods at all three tested model sizes and ranks first on TextWorld and ScienceWorld at each size. ALFWorld experiments support the learned concept's contribution beyond prior-only scoring and show further gains when combined with Reflexion. Code is available at https://anonymous.4open.science/r/CB-Agent-DFF9/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, large language model agents, concept-based learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S4rPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CB-Agent extends concept-based in-context learning (CB-ICL) to sequential action selection for interactive language agents. It extracts environment-action decisions from expert trajectories and fits a single concept vector by regularized maximum likelihood, modeling the conditional distribution over candidate actions given the context with an exponential-family model that combines the learned concept with frozen language-model features and a length-adjusted prior.

The method is evaluated on TextWorld, ALFWorld, and ScienceWorld with Qwen3-4B/8B/14B, achieving the highest cross-benchmark average at all three sizes, along with an ALFWorld control isolating the concept's contribution and a separate combination with Reflexion.

Critical strengths
  1. The method is simple, principled, and broadly validated. Only a lightweight concept vector is fitted (no LLM parameters are updated), the exponential-family formulation is well-motivated, and the gains are consistent across three benchmarks and three model sizes.

  2. The θ=0 control provides clean causal evidence for the method's core claim.

Critical weaknesses
  1. Performance is uneven and the headline metric partly obscures this. CB-Agent does not lead on ALFWorld and the paper does not diagnose why, beyond a brief note that task-level success may not reflect individual action improvements. The "highest cross-benchmark average" framing gives a slightly stronger impression than the per-benchmark picture warrants.

  2. The supervision signal's quality is not established. The "expert" is a larger model (Qwen3-32B) whose own success rate is never reported as an upper bound, and it is not stated whether expert trajectories are filtered to successful completions. Without this, it is hard to judge how much room remains, or whether the fitted concept encodes suboptimal behavior.

  3. The released code is incomplete and difficult to read, which hampers reproducibility.

I would be willing to raise my assessment to weak accept if the concerns above are adequately addressed.

Questions for authors
  1. Please report Qwen3-32B's own success rate on these benchmarks as an upper bound. How much does CB-Agent lose relative to directly deploying the expert model, and does the efficiency gain justify that gap?

  2. Why does CB-Agent not lead on ALFWorld despite the large θ=0 gains?

  3. The released code appears incomplete and difficult to read. Can you provide a complete, well-documented codebase that reproduces the reported numbers?

3 comments

Thank you for the constructive assessment and for identifying the expert-reference, ALFWorld and reproducibility questions. We now provide expert performance and clarify the supervision protocol and the scope of the claims.

Q1. Expert performance and supervision quality

Thank you for requesting this reference. We compare the Qwen3-32B expert with Qwen3-14B CB-Agent on all three benchmarks below.

Benchmark32B Expert14B CB-AgentGap
TextWorld89.782.37.4
ALFWorld82.881.31.5
ScienceWorld73.164.19.0

All scores are on a 0–100 scale: ALFWorld reports success rate, while TextWorld and ScienceWorld report normalized scores. Gaps are score differences, corresponding to percentage-point differences on ALFWorld.

The 14B CB-Agent comes within 1.5 percentage points of the 32B expert on ALFWorld, using a smaller frozen backbone and fitting only a lightweight concept vector. The TextWorld and ScienceWorld results quantify the remaining performance gaps. We also clarify that only successfully completed expert trajectories are retained for concept fitting; failed trajectories are excluded. We will include the expert comparison and trajectory-filtering protocol in the revised manuscript.

Where local resources were insufficient to deploy Qwen3-32B, we used API access for expert evaluation. Since these serving conditions differ from those of the locally deployed smaller agents, their wall-clock times are not directly comparable. We clarify a key deployment advantage of CB-Agent: once the concept is fitted, the agent reuses expert experience using only the smaller backbone and the learned vector, without deploying or repeatedly querying the large expert model during inference.

Q2. Why does CB-Agent not lead on ALFWorld despite the α=0 gains?

The two comparisons address different questions. The α=0 control isolates the concept's contribution within the same pipeline, yielding gains of 10.4, 6.0 and 14.9 percentage points. The main table compares complete systems, where StateAct and ReflAct also introduce mechanisms for state tracking and reflection.

ALFWorld has relatively structured task routines, and strong baselines already achieve high success rates. We hypothesize that this leaves less room for improved action scoring alone: remaining failures may depend more on planning and state tracking, so better candidate selection need not translate into higher complete-task success. The concept can nevertheless complement feedback-based interaction: under the same two-trial budget, CB-Agent + Reflexion improves over Reflexion by 4.5, 7.5 and 5.2 percentage points (Table 2). CB-Agent also leads on TextWorld, ScienceWorld and the three-benchmark average at all three evaluated Qwen3 sizes, further supporting its effectiveness.

We will clarify this interpretation as a hypothesis and emphasize the per-benchmark results alongside the cross-benchmark average.

Q3. Complete, documented code

Thank you for pointing this out. We have updated the repository with the complete codebase, including fixed evaluation configurations, dependencies, and usage instructions to support reproduction of the reported results.

I thank the authors for their detailed response. The rebuttal addressed some of my concerns; however, after considering the other reviewers' feedback as well, I believe the current score is consistent with my overall assessment of the paper, and I therefore maintain my score.

Thank you for taking the time to read our response and consider the new results. We’re glad the rebuttal addressed some of your concerns, and we’ll include all the additional experimental results in the final manuscript.

Could you let us know which concerns remain unresolved? We’d be happy to address them directly and clarify any points that would help with your final assessment. With only three hours left in the discussion period, we may have limited time to respond to your new feedback. If the discussion closes before we can reply, we’ll use your new feedback to revise the final manuscript and explicitly discuss any remaining limitations.

Reviewer_eQ3LOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CB-Agent trains a numerical action evaluator for each task type based on an expert’s action history, while keeping the language model’s core fixed. It combines the linear scores learned for the fixed features with a length-normalized language model prior distribution, and uses this distribution within ReAct to select action candidates. The demonstration history remains available during the inference and path planning stages but is excluded from the inputs to the action evaluator. It reports the highest cross-benchmark average across three Qwen3 scales and three benchmarks, and on ALFWorld, it provides a comparison with a control group that uses only the prior distribution, as well as combined experiments with Reflexion that include two separate trials.

Critical strengths
  1. ALFWorld’s controlled experiments directly demonstrate the effectiveness of learned scores. When all other settings are held constant, the success rate increases by 10.4, 6.0, and 14.9 percentage points. This improvement is observed relative to the prior-only control—which does not use ReAct but disables learned scores—and serves as evidence that learning scores during the same action selection process is beneficial.

  2. When combining the CB-Agent with Reflexion in ALFWorld, both methods achieve success rates that are 4.5, 7.5, and 5.2 percentage points higher than Reflexion under conditions using two attempts. This supports their usefulness when used alongside existing agent methods.

Critical weaknesses
  1. Since the evaluated models were the three sizes of Qwen3, it has not yet been confirmed whether the improvements hold across other model families. Control experiments designed to isolate the effect of the learned scores are also limited to ALFWorld. In ScienceWorld, CB-Agent samples 20 responses to generate candidates, but no prior-only control experiment—which would isolate the effect of the learned scores in the same process—has been reported. The authors acknowledge these limitations, which restrict the scope to which the results can be generalized.

  2. The difference from the best baseline is small in some settings. For example, the difference in two ScienceWorld settings is 0.6 points, but variability due to repeated runs was not reported. The manuscript does not specify the number of evaluation tasks for TextWorld and ScienceWorld, nor does it sufficiently explain the candidate generation settings for the baselines. Consequently, uncertainty remains regarding the magnitude of improvement and the model’s stability.

Questions for authors
  1. Could you compare the prior-only control (alpha=0) using a single representative backbone while maintaining the rest of the process in both TextWorld and ScienceWorld? It would also be helpful if you could explain how many responses the ScienceWorld baseline samples, as this would help distinguish the contribution of the learned scores.

  2. Could you evaluate one representative backbone from another applicable model family on the benchmark and compare it with ReAct, which uses the same backbone? This would help us determine whether the improvements hold true beyond Qwen3.

  3. Could you please provide the number of evaluation tasks for TextWorld and ScienceWorld, and present the mean, standard deviation, and number of independent runs from repeated comparisons between CB-Agent and the best baseline for each setting where the difference is small? In particular, examining the two ScienceWorld settings that show a 0.6-point difference would help us assess the stability of the improvement.

3 comments

Thank you for recognizing the controlled ALFWorld results and the complementary gains with Reflexion. Following your suggestions, we extended the prior-only control to both remaining benchmarks, evaluated additional model families, and added three-seed results. We will incorporate these additional results and the clarified evaluation settings into the revised manuscript.

Q1. Learned-score contribution and ScienceWorld sampling

We extended the alpha=0 control to TextWorld and ScienceWorld at all three Qwen3 sizes. Only the learned concept term is disabled; the remaining pipeline, including reasoning/routing, candidate generation, the language-model prior, length correction and interaction budget, is unchanged. The full CB-Agent scores below are reproduced from the original Table 1; the prior-only evaluations are newly added.

TextWorld

Configuration4B8B14B
α=023.121.422.1
CB-Agent63.577.582.3
Gain40.456.160.2

ScienceWorld

Configuration4B8B14B
α=053.652.657.3
CB-Agent59.361.664.1
Gain5.79.06.8

All scores and gains are on the original 0–100 benchmark scale. These comparisons extend the evidence for the learned concept beyond ALFWorld to both environment-supplied and model-generated action candidates. In particular, the ScienceWorld gains remain when the candidate-generation budget is held fixed, separating the learned-score contribution from the benefit of candidate proposal itself.

For clarity, ScienceWorld ReAct and ExpeL each generate one response per decision at temperature 0. CB-Agent and its prior-only control generate 20 independently sampled candidate responses at temperature 0.7, as specified for CB-Agent in Appendix B (p. 17). Methods that inherently require multiple samples, such as Self-Consistency, use temperature 0.7 on all benchmarks; other baselines use temperature 0. CB-Agent uses temperature 0 on TextWorld and ALFWorld. We will make these method-specific settings explicit. The complete-system comparison uses different sampling budgets, while the matched-budget alpha=0 control isolates the contribution of the learned concept.

Q2. Evaluation beyond Qwen3

We additionally compared ReAct and CB-Agent on 134 ALFWorld tasks using Gemma 3 instruction-tuned backbones, retaining the same hyperparameters according to the additional experiment protocol. The Gemma 3 results are:

BackboneReAct solvedCB-Agent solvedReAct SRCB-Agent SRDifference
Gemma 3 4B IT14/13449/13410.4%36.6%+26.1 pp
Gemma 3 12B IT58/13472/13443.3%53.7%+10.4 pp

Improvements of 26.1 and 10.4 percentage points demonstrate that gains are not confined to Qwen3.

Q3. Evaluation size and repeated-run variability

TextWorld evaluates 160 instances (16 task types, 10 instances each), and ScienceWorld evaluates 150 instances (30 task types, 5 instances each). Because each task type has equal instance coverage, averaging per-type instance means and averaging all instance scores are equivalent.

We added three-seed evaluations of CB-Agent, ReAct and ExpeL. The tables report mean ± standard deviation across the three runs. Across seeds, demonstration order is shuffled. For ReAct and ExpeL, temperature is 0, so seed-dependent variation comes from the changed demonstration order. For CB-Agent on TextWorld, demonstration ordering affects ReAct reasoning/routing at temperature 0; ScienceWorld additionally has candidate sampling at temperature 0.7. Thus, temperature-0 baselines can vary because their input prompts differ across runs.

ScienceWorld:

Method4B8B14B
ReAct55.39 ± 1.6856.85 ± 1.6363.03 ± 1.54
ExpeL59.02 ± 0.7758.70 ± 0.4363.41 ± 2.58
CB-Agent60.06 ± 0.6761.30 ± 1.2164.71 ± 0.55

ExpeL is the strongest original ScienceWorld baseline at each size. CB-Agent's mean gains over ExpeL are 1.04, 2.60 and 1.30 points. The 4B and 14B results directly address the two original 0.6-point differences: CB-Agent retains higher mean scores in the repeated evaluations, with run-to-run variability now explicitly reported.

TextWorld:

Method4B8B14B
ReAct61.88 ± 1.4175.00 ± 0.8679.47 ± 1.89
ExpeL40.25 ± 3.1946.27 ± 0.9136.30 ± 2.63
CB-Agent63.85 ± 0.8877.31 ± 0.4282.38 ± 0.36

CB-Agent also has higher mean scores than ReAct at all three TextWorld sizes. The TextWorld repetitions compare CB-Agent with ReAct and ExpeL. We will report these three-seed summaries separately from the original results and the new alpha=0 controls.

I appreciate your detailed reply and the additional experiments you ran. They answered all of my questions:

  1. On TextWorld and ScienceWorld, the alpha=0 control demonstrated the improvement from learning the concept while holding the remainder of the pipeline constant. This includes when the same 20 sample budget was used as ReAct/ExpeL on ScienceWorld.
  2. The Gemma 3 results on ALFWorld demonstrate the improvements do not only apply to Qwen3.
  3. The number of tasks and 3 seed averages demonstrate that CB-Agent maintains a greater mean in situations where there was little difference originally.

I have increased my rating to 3. There are two minor issues I would like addressed in the final version; please make it clear that the full ScienceWorld comparison uses 20 samples per task for CB-Agent, as opposed to 1 per task for ReAct and ExpeL. Additionally, please acknowledge that some margins remain relatively small (e.g., 1.3 over ExpeL at 14B, which is within ExpeL's standard deviation of 2.58).

Thank you for reviewing our additional experiments and updating your assessment. We are glad to hear that the new results "answered all of your questions".

We will incorporate both of your remaining suggestions, along with all additional experimental results from the rebuttal, into the final manuscript.

Reviewer_2L5KOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CB-Agent, which learns a concept vector from expert action choices in demonstration trajectories and combines it with frozen language-model features and prior probabilities to score candidate actions. The scorer is integrated into ReAct and reused throughout interaction without updating the language model.

Critical strengths

1.Directly using expert actions to supervise candidate selection is intuitive, and the training objective is clearly defined.

2.Experiments cover three interactive environments and three model sizes. The ALFWorld control experiments support the contribution of the concept vector, and combining the method with Reflexion yields further improvements.

Critical weaknesses
  1. Equations (3) and (6) essentially train a linear action scorer on frozen model features and combine it with a language-model prior. The paper should clarify what distinctive advantages this “concept” formulation offers over conventional lightweight supervised action reranking.

  2. Algorithm 1 still uses demonstration-conditioned ReAct for reasoning and think-or-act routing at each step, with additional candidate generation and scoring on the action branch. Therefore, excluding demonstrations from the action-scoring stage does not directly imply reduced computational overhead for the overall system.

  3. The α=0\alpha=0 control, which removes the contribution of the concept vector, is evaluated only on ALFWorld, whereas the method’s main advantages over the baselines are observed on TextWorld and ScienceWorld.

Questions for authors

Weaknesses

1 comment

Thank you for recognizing the direct supervision from expert actions and the supporting ALFWorld controls. We address the method's positioning, computational scope and the requested cross-benchmark controls below.

W1. What does the concept formulation contribute beyond supervised reranking?

Thank you for asking us to clarify this distinction. Although action selection takes the form of candidate reranking, CB-Agent provides a theoretically motivated framework for representing and reusing expert experience. Following CB-ICL, the concept represents task-specific preferences in a frozen LLM feature space. We extend this perspective to sequential decisions through a prior-weighted exponential-family model: Appendix D motivates the exponential form through a local information-geometric argument, while Appendix C connects the decision-level fitting objective to trajectory likelihood under the stated conditions and establishes a unique optimum for the regularized objective. Thus, the formulation connects task representation, probabilistic action selection, and learning from trajectories, beyond specifying a scoring head.

The empirical evidence complements this foundation. Figure 2 shows that concepts estimated from different trajectory samples of the same task type tend to cluster, supporting their interpretation as shared task-related representations. The expanded alpha=0 controls across all three benchmarks demonstrate the learned adjustment's contribution beyond the model prior. Improvements when combined with Reflexion further show that the fixed concept remains useful as feedback updates the interaction context.

Our contribution is therefore the principled extension of concept-based learning to sequential action selection, supported by both objective analysis and empirical validation.

W2. Computational overhead of the complete agent

Thank you for highlighting this distinction. Our claim concerns how expert experience informs action selection, rather than reduced end-to-end computation. As stated in Appendix F, candidate generation and scoring introduce additional computation, and demonstrations remain in ReAct reasoning and routing.

On the non-Think branch, CB-Agent selects the executed action through a concept-conditioned probability model. Expert experience enters action scores through a reusable learned concept, without reinserting expert trajectories into the scoring context. This reduces the scorer's reliance on retrieving and applying demonstration information from long prompts, while retaining the current interaction history for state-dependent decisions.

The contribution is therefore a structured, theoretically motivated mechanism for using expert experience in action selection. The expanded prior-only controls demonstrate its performance benefit while keeping the remaining pipeline unchanged.

W3. Prior-only controls on TextWorld and ScienceWorld

We have added the requested controls at all three model sizes, disabling only the learned concept term while keeping the rest of the pipeline unchanged. Full CB-Agent scores are reproduced from original Table 1; alpha=0 scores are newly evaluated.

TextWorld

Configuration4B8B14B
α=023.121.422.1
CB-Agent63.577.582.3
Gain40.456.160.2

ScienceWorld

Configuration4B8B14B
α=053.652.657.3
CB-Agent59.361.664.1
Gain5.79.06.8

Together with original Table 3 on ALFWorld, these results support the contribution of the learned concept on all three benchmarks, including the two where the main-table advantages were concentrated. The ScienceWorld control retains the same 20-response candidate-generation procedure, showing that the gains come from the learned concept rather than additional candidate sampling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.