- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes CILC, a method to mitigate latent confounding bias in LLMs by intervening on responsible neurons. CILC infers a latent representation of the input with a VAE aiming to encode the latent confounding, then maps it into soft prompts, identifies the neurons most sensitive to these soft prompts with integrated gradients attribution, and scales down their activations at inference time. The method is motivated by the argument that front-door-based debiasing methods are invalid when latent confounders also affect the reasoning process. Experiments on recommendation tasks (on top of TALLRec and CoLLM) and reasoning benchmarks (on top of ACPS) show consistent accuracy improvements.
Critical strengths
- The paper is very well written and easy to follow. Fig. 2 is well crafted.
- Terminology is precise, and notation is clear.
- The motivation is convincing: latent confounders encoded in the model parameters can also affect the reasoning process, which violates the assumptions of front-door adjustment.
Critical weaknesses
Major
-
W1. The main text only reports accuracy. The RQ3 and an important causal analysis is all deferred to the appendix (Sec. 4.3), as is the related work. Studies beyond accuracy are important to support the main claim, so in my opinion they belong in the main text, and the appendix should only complement them. I understand these analyses could be moved to the main text if the extra camera-ready page is allowed, but the content was not well organized within the 9-page limit at submission time.
-
W2. It seems to me the paper misses convincing evidence that the improvement comes from mitigating latent confounding. The ablations (Tab. 4-5) only change which neurons are intervened on, while the rest of the pipeline is kept fixed. They show that the BIG-selected neurons matter, but not that the latent confounder does. Without such controls, the gains could equally come from other elements in the pipeline. Some results also raise some concerns:
- In Fig. 3, the AUC at γ = 1 (i.e., the intervention leaves the activations unchanged) is about 0.686 on movie and 0.580 on book, well above TALLRec (0.672 and 0.564, Tab. 1). Unless I am missing something, most of the gain would then not come from the intervention. Please correct me if I misunderstood this point.
- The ATE is measured on the injected soft prompt, which is exactly what the neurons are selected for, so a positive ΔATE (Tab. 6) is somewhat expected by construction. Moreover, even for inputs without the latent proxy, the intervention increases the average predicted probability by about 0.05-0.06 (Tab. 7), i.e., more than twice the effect of the proxy itself (about 0.02). This suggests that the intervention changes the model's predictions well beyond the confounding effect it targets, right?
-
W3. A synthetic controlled experiment is missing. The core claim is that CILC addresses the failure of front-door adjustment, yet CILC is never compared with front-door methods in a very controlled synthetic experiment where one can clearly see front-door failing and the proposed method succeeding. Given the length of the pipeline (VAE, soft-prompt mapping, BIG, intervention), a synthetic experiment with a known latent confounder affecting the mediator would help convince the reader.
-
W4. Uncertainties are missing. All experiments are repeated over five seeds (Sec. 4.1), but only means are reported. Standard deviations or statistical tests are needed to assess whether the gains are significant.
Minor
- W5. The interpretation of γ in App. E.3 seems reversed: since the intervention sets , a small γ corresponds to a strong suppression, not an insufficient one.
Questions for authors
- Q1. Could the authors expand on the choice of a VAE to extract the input representation?
- Q2 (related to W2) Could the authors demonstrate that the improvements originate from the effect of the latent confounders?
- Q3 (related to W2). Is it possible, at least in principle, to design a synthetic experiment where the confounder is known? How would the proposed method compare to prior in this setting? What could be measured?
Summarizing, while I find the paper well presented and well motivated, the lack of evidence that the improvements stem from mitigating latent confounding prevents me from recommending acceptance.