- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes ReLoC, which learns agent-local response models through interventions and composes them along a known communication DAG to predict intermediate states and final answers under unseen interventions. Each message consists of a task label and three Boolean concepts. Observed inputs use empirical response frequencies, while unobserved inputs use a Laplace-smoothed factorized count estimator, augmented with joint features when matched controls reveal interactions. Acquisition proceeds recursively upstream from the outcome agent and allocates probes using coverage gaps and prediction errors. Experiments cover three tasks, three topologies, and four LLMs, evaluating local response, propagation, trajectory, and final-answer fidelity under a budget of thirty-two agent calls per case.
Critical strengths
-
The objective is meaningful. Learning reusable local response models could reduce the execution cost of answering multiple joint-intervention queries, beyond measuring isolated intervention effects. The known communication graph and direct local interventions provide a clear modeling interface.
-
The evaluation distinguishes outcome prediction from process reproduction. Reporting local response, propagation, trajectory, and final-answer fidelity, together with changed-versus-unchanged outcomes, exposes mechanism errors that aggregate answer fidelity would otherwise conceal. This is a valuable aspect of the experimental design.
Critical weaknesses
-
High aggregate answer fidelity does not establish reliable causal-mechanism explanations, and the incremental contribution of recursive acquisition is limited against the most relevant baseline. On Qwen3-4B, Invariant already achieves 93.3% versus ReLoC's 93.7%; on Llama, the corresponding figures are 84.4% and 85.6%. Fidelity on genuinely changed outcomes is only 44.8% pooled and 31.5% on Qwen. Much of the large advantage over Global LLM therefore reflects avoiding false changes rather than accurately capturing intervention propagation. The explanatory evidence is weaker still: trajectory exact match is only 19.6--50.8%, and 41.1--77.5% of correctly predicted final answers contain incorrect intermediate trajectories. These results do not establish reliable mechanism explanations. Full-local, which shares the estimator and composition rule, is the more direct test of the acquisition contribution. ReLoC is slightly worse on Hy3 and 1.2 points worse on Llama, leaving a consistent advantage of recursive error-guided acquisition unestablished.
-
Compositional generalization relies on strong restrictions and insufficiently validated extrapolation, while the proposed future directions correspond to central current gaps. Communication is restricted to a label and three Boolean fields, with a fixed known DAG, predefined intervention catalogs, and case-specific fitting. For unseen inputs, the factorized count estimator uses a conditional-independence approximation; interactions are added only after relevant errors and matched controls are observed. No guarantee establishes that the finite acquisition budget discovers important interactions or controls composition error. Passing the local mode at each step also differs from marginalizing stochastic mechanisms. Empirically, 73.4% of queries require statistical extrapolation, whose fidelity on CEBaB is only 48.2%. Targeting changed intermediate states, free-form messages, and dynamic communication are deferred to future work. However, coverage of changed states already drives current missed effects and directly limits the existing generalization claims. The evidence currently supports an empirical surrogate on a restricted discrete interface, rather than a broadly reliable global causal explanation.
Questions for authors
-
Which improvements over Full-local have statistical support, and which do not? Please report paired differences separately for changed and unchanged outcomes to establish the contribution of recursive acquisition independently of the large gap to Global LLM.
-
When the final answer is correct but the intermediate trajectory is wrong, does the model still identify the agents or concepts responsible for the outcome? What independent evidence supports mechanism explanation beyond endpoint prediction?
-
Under what structural conditions is extrapolation to unseen input combinations valid? How are unreliable predictions identified when matched controls are unavailable or higher-order interactions remain unobserved? For stochastic responses, how much error does local-mode composition introduce relative to probabilistic marginalization?
-
Acquisition targeting changed intermediate states is proposed as future work, but this corresponds directly to the current dominant failure mode. Under the same budget, how does ReLoC compare with simple minimum-coverage or changed-state-oriented acquisition policies? If free-form messages and dynamic communication remain unsupported, how should the current conclusions be explicitly restricted to four-field messages and fixed DAGs?