- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CB-Agent extends concept-based in-context learning (CB-ICL) to sequential action selection for interactive language agents. It extracts environment-action decisions from expert trajectories and fits a single concept vector by regularized maximum likelihood, modeling the conditional distribution over candidate actions given the context with an exponential-family model that combines the learned concept with frozen language-model features and a length-adjusted prior.
The method is evaluated on TextWorld, ALFWorld, and ScienceWorld with Qwen3-4B/8B/14B, achieving the highest cross-benchmark average at all three sizes, along with an ALFWorld control isolating the concept's contribution and a separate combination with Reflexion.
Critical strengths
-
The method is simple, principled, and broadly validated. Only a lightweight concept vector is fitted (no LLM parameters are updated), the exponential-family formulation is well-motivated, and the gains are consistent across three benchmarks and three model sizes.
-
The θ=0 control provides clean causal evidence for the method's core claim.
Critical weaknesses
-
Performance is uneven and the headline metric partly obscures this. CB-Agent does not lead on ALFWorld and the paper does not diagnose why, beyond a brief note that task-level success may not reflect individual action improvements. The "highest cross-benchmark average" framing gives a slightly stronger impression than the per-benchmark picture warrants.
-
The supervision signal's quality is not established. The "expert" is a larger model (Qwen3-32B) whose own success rate is never reported as an upper bound, and it is not stated whether expert trajectories are filtered to successful completions. Without this, it is hard to judge how much room remains, or whether the fitted concept encodes suboptimal behavior.
-
The released code is incomplete and difficult to read, which hampers reproducibility.
I would be willing to raise my assessment to weak accept if the concerns above are adequately addressed.
Questions for authors
-
Please report Qwen3-32B's own success rate on these benchmarks as an upper bound. How much does CB-Agent lose relative to directly deploying the expert model, and does the efficiency gain justify that gap?
-
Why does CB-Agent not lead on ALFWorld despite the large θ=0 gains?
-
The released code appears incomplete and difficult to read. Can you provide a complete, well-documented codebase that reproduces the reported numbers?
3 comments
I thank the authors for their detailed response. The rebuttal addressed some of my concerns; however, after considering the other reviewers' feedback as well, I believe the current score is consistent with my overall assessment of the paper, and I therefore maintain my score.
Thank you for taking the time to read our response and consider the new results. We’re glad the rebuttal addressed some of your concerns, and we’ll include all the additional experimental results in the final manuscript.
Could you let us know which concerns remain unresolved? We’d be happy to address them directly and clarify any points that would help with your final assessment. With only three hours left in the discussion period, we may have limited time to respond to your new feedback. If the discussion closes before we can reply, we’ll use your new feedback to revise the final manuscript and explicitly discuss any remaining limitations.
Thank you for the constructive assessment and for identifying the expert-reference, ALFWorld and reproducibility questions. We now provide expert performance and clarify the supervision protocol and the scope of the claims.
Thank you for requesting this reference. We compare the Qwen3-32B expert with Qwen3-14B CB-Agent on all three benchmarks below.
All scores are on a 0–100 scale: ALFWorld reports success rate, while TextWorld and ScienceWorld report normalized scores. Gaps are score differences, corresponding to percentage-point differences on ALFWorld.
The 14B CB-Agent comes within 1.5 percentage points of the 32B expert on ALFWorld, using a smaller frozen backbone and fitting only a lightweight concept vector. The TextWorld and ScienceWorld results quantify the remaining performance gaps. We also clarify that only successfully completed expert trajectories are retained for concept fitting; failed trajectories are excluded. We will include the expert comparison and trajectory-filtering protocol in the revised manuscript.
Where local resources were insufficient to deploy Qwen3-32B, we used API access for expert evaluation. Since these serving conditions differ from those of the locally deployed smaller agents, their wall-clock times are not directly comparable. We clarify a key deployment advantage of CB-Agent: once the concept is fitted, the agent reuses expert experience using only the smaller backbone and the learned vector, without deploying or repeatedly querying the large expert model during inference.
The two comparisons address different questions. The α=0 control isolates the concept's contribution within the same pipeline, yielding gains of 10.4, 6.0 and 14.9 percentage points. The main table compares complete systems, where StateAct and ReflAct also introduce mechanisms for state tracking and reflection.
ALFWorld has relatively structured task routines, and strong baselines already achieve high success rates. We hypothesize that this leaves less room for improved action scoring alone: remaining failures may depend more on planning and state tracking, so better candidate selection need not translate into higher complete-task success. The concept can nevertheless complement feedback-based interaction: under the same two-trial budget, CB-Agent + Reflexion improves over Reflexion by 4.5, 7.5 and 5.2 percentage points (Table 2). CB-Agent also leads on TextWorld, ScienceWorld and the three-benchmark average at all three evaluated Qwen3 sizes, further supporting its effectiveness.
We will clarify this interpretation as a hypothesis and emphasize the per-benchmark results alongside the cross-benchmark average.
Thank you for pointing this out. We have updated the repository with the complete codebase, including fixed evaluation configurations, dependencies, and usage instructions to support reproduction of the reported results.