- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether language models can compose newly injected facts when training presents each fact separately. It constructs 300 counterfactual two-hop chains across five domains and distinguishes atomic-fact acquisition from successful composition. The proposed method, BRACE, attaches descriptors drawn from a shared pool to the bridge entity in both hops, creating a common representation across otherwise separate training examples. Experiments cover four models from two families, ranging from 1.7B to 14B parameters. BRACE achieves the highest composed-accuracy point estimate in every model–domain combination. Further analyses examine descriptor sharing, descriptor correctness, intermediate bridge predictions, chain-of-thought prompting, and the ordering of composition demonstrations.
Critical strengths
- The benchmark separates atomic acquisition from composition and controls for answers derived from real-world knowledge.
- BRACE achieves the highest composed-accuracy point estimate in all 20 model–domain combinations. The descriptor controls and training-budget comparisons help interpret these gains.
Critical weaknesses
- Composition rate uses a different acquired-chain subset for each method. Differences in subset difficulty can therefore influence the comparison. The smaller denominators reported in Appendix H leave this issue unresolved.
- The benchmark uses entity-disjoint two-hop chains trained together. Shared entities and sequential revisions would provide a stronger test of the proposed continual-updating application.
Questions for authors
- How does the composition-rate advantage change on chains acquired by both methods and across acquisition thresholds?
- Does the descriptor-sharing gain persist across randomized descriptor partitions and sequential updates with reused entities?